PDF extraction is reconstruction, not reading: budget the pipeline that way

A walkthrough published on dev.to steps through what pypdf and PyMuPDF actually do when they open a file, and its central point is structural rather than a matter of tool preference: a PDF has no concept of a paragraph, a column or a table, and does not even store text in reading order ^[1]. If that is true, then every ingestion pipeline that outputs paragraphs and table cells is running a reconstruction, and it should be scoped, budgeted and tested as one rather than as a library call.

The reason is in the format. According to the dev.to piece, a PDF is essentially a set of vector drawing instructions for a printer canvas ^[2], and when a word processor exports to PDF it discards the semantic document structure in favour of visual position ^[3]. Instead of a heading or table element, the encoder writes low-level operators: begin a text object, set a font at a size, move the cursor to an absolute coordinate, draw a character stream, end the object ^[4]. What is absent from that stream is space characters, paragraph breaks and reading order ^[5]. Because glyphs land at hardcoded x,y points, a generator may draw the footer first, the header second, the right-hand sidebar third and the main paragraph last ^[6], and a naive dump of the raw stream produces interleaved columns ^[7].

So the work splits into two jobs, which the article labels as decoding binary anatomy with pypdf and then spatial geometry and layout analysis with PyMuPDF ^[20]. The first job is plumbing with sharp edges. pypdf's reader seeks to the end of the file, reads the last 1,024 bytes to find the startxref keyword, and parses the cross-reference table into an index from object IDs to byte offsets ^[8]. The trailer dictionary points at the document catalog, at metadata such as Author, CreationDate and Producer, and at the total XRef entry count ^[9]; the root catalog exposes keys including /Pages, /Names and /AcroForm ^[10]. Page drawing commands live in a stream, or an array of streams, under /Contents ^[11], usually compressed with FlateDecode and inflated in RAM ^[12]. Stream strings are not UTF-8 by default but 8-bit encodings, often WinAnsiEncoding or PDFDocEncoding, which is why latin1 is used as a decode that preserves byte values instead of raising ^[13]. The mapping from a stream alias like /F1 to real characters sits in the font dictionary, with /BaseFont names and embedded /ToUnicode character maps ^[14].

The second job is where the error rate lives. PyMuPDF runs MuPDF's C engine ^[15] and returns, for each word, an eight-element tuple: bounding box, the word text, and block, word and line numbers ^[16], with (x0,y0) top-left and (x1,y1) bottom-right ^[17]. Four of those eight fields are geometry, one is text, and three are grouping indices ^[1], and the grouping exists because the snippet clusters tokens spatially to solve multi-column reading order ^[18]. That is a heuristic with a failure mode, not a field read out of the file.

Practical consequence: treat column detection, header and footer stripping, and table cell assignment as estimators. Keep a fixture corpus of the actual vendor documents you receive, assert on extracted output rather than on exit codes, and price the long tail into the schedule.

Watch three things in your own corpus. Files whose fonts ship no /ToUnicode map ^[14], since character recovery depends on that mapping. Files carrying /AcroForm data ^[10], where the values you want are form fields rather than drawn text. And pages leaning on /XObject entries, which the article notes hold embedded images or sub-canvases ^[19] and therefore push content outside the main stream you are parsing.