The codec runs last: what a hand-written ORC parser shows about columnar speed

A developer wrote a small ORC parser from scratch, generated a deliberately boring 5-million-row orders table, and read the file by hand instead of repeating the usual summary about columnar storage and predicate pushdown ^[1]. What comes out of the byte-level walk is a reordering of credit: the compression codec only runs after encoding is done, and by then most of the size reduction has already happened ^[17][21]. Start with the read path. The last byte of an ORC file holds the length of the PostScript, and in this file the final 26 bytes decode as a Protocol Buffers message ^[7]. That means a query engine performs three reads, the last byte, the PostScript, then the footer plus metadata, totalling a few hundred bytes against a 53 MB file ^[8]. After those reads it has the schema, the row count, every stripe's byte offset, and min/max statistics for every column, without having touched a row ^[9]. The ordering is forced rather than clever: the codec name lives in the PostScript, and the PostScript cannot itself be compressed, because you would need the codec to decompress the thing that names the codec ^[10]. Locality is equally mechanical. Inside a stripe a column is a set of streams laid out contiguously, so the country column appears as 23 bytes of DICTIONARY_DATA, 5 bytes of LENGTH and 310,404 bytes of DATA, while price DATA is 2,374,052 bytes ^[14]. Reading country from one stripe is a single range read rather than 687,104 scattered seeks ^[15]. Nullability is priced the same way: the PRESENT stream is omitted entirely when a column has no nulls, so null tracking here cost zero bytes, where a row format pays on every row whether you use it or not ^[16]. The encoding numbers are the part worth arguing about. Five million sequential 64-bit integers occupy 11 KB, because ORC's RLE v2 picked its DELTA sub-encoding and stored a base, a delta width and a run length ^[17]. Raw, those values are 40 MB, so the encoder cut roughly 3,600x before zlib was invited in ^[18]. Sort order is the lever: the same 5 million dates take 7,390,184 bytes in random order and 4,898 bytes sorted, which the author measures as 1,500x ^[19]. Compression that only works when neighbours are similar means physical row order is a schema decision, not a cosmetic one. Two defaults deserve attention. PyArrow's dictionary_key_size_threshold defaults to 0.0, which disables dictionary encoding outright; setting it to 0.8 was worth 14% of the file from one keyword argument ^[20][4]. And the codec runs in independent 64 KB chunks with a 3-byte header, where bit 0 marks a chunk as stored raw if compressing it made it bigger, which is also what makes seeking cheap: one large compressed blob would mean inflating 8 MB to read the last value ^[21][22]. Read the setup before borrowing the conclusions. Stripe size was set to 8 MB on purpose, because production ORC uses 64 to 256 MB stripes and a default stripe would have swallowed this whole 53 MB dataset, leaving nothing to skip ^[6][4]. The skip machinery is cheap at this scale: 8,412 bytes of index against 8,544,070 bytes of data in stripe 0, which the author puts at 0.1% overhead ^[12][13], with min/max statistics kept at three zoom levels, 448 bytes for the file, 663 for a stripe, and 8.4 KB per stripe for row groups of 10,000 rows ^[24]. Uncompressed ORC is still 3.6x smaller than the equivalent 310 MB CSV, a gap the author attributes purely to layout and encoding ^[23][5]. Worth checking on your own writers: whether dictionary encoding is switched on at all, whether the columns you filter by are the columns you sort by, and whether your stripe size leaves more than one stripe to skip.