Iceberg parks its kilobyte statistics in a Puffin sidecar the planner opens on demand

Take the delete case at its own numbers. Three hundred rows out of four million is 0.0075 percent of the file, about one row in 13,333 ^[20]. In format version 2 that edit lands as a Parquet position delete file naming the data file and the position of each deleted row ^[4]. The write is cheap. The read path is where it settles, because every later read of that data file has to open the delete file, decode Parquet, build a set of positions, and filter ^[5]. Do that across ten thousand data files and, per the article, delete handling dominates query time ^[5].

For a long time my answer to delete-heavy tables was to compact more often. That works by rewriting the data files, which is a full write issued to avoid a per-read decode. The version 3 deletion vector goes at the decode instead: a bitmap blob fetched by offset, described in the footer rather than parsed out of a second Parquet file ^[15][16]. Whether that trade favours you depends on your read-to-write ratio, and the source gives no number for either.

The join case is the same mismatch pointed the other way. The planner has row counts, bounds, and null counts for every column in every file ^[2], which tells it how large the scan is and nothing about how selective the join key is. Two billion rows against forty thousand is a 50,000-to-1 ratio ^[19]; the small side looks obviously broadcastable right up to the point where the optimizer wants the distinct count on the fact side, does not find it, guesses, and shuffles terabytes a broadcast join would have avoided ^[1][3].

You cannot fix that by adding a column to the manifest. Per-file distinct counts do not sum, because the same value turns up in many files ^[9]. What does merge is a sketch, and a mergeable sketch is thousands of bytes ^[10] against manifest metrics that run a few bytes per column per file ^[8]. The planner reads those entries on every query ^[7]. Anything you park there, every query carries.

So the container is deliberately dull: four magic bytes, 0x50 0x46 0x41 0x31, blobs written back to back, and a JSON footer saying what each blob is, what it was computed for, and where it sits ^[14][15]. The footer is the index, which is what makes reading one statistic without reading the file possible, and random access was one of the goals stated when the format was proposed ^[13]. The magic spells PFA1, for Fratercula arctica version 1 ^[17]. Someone enjoyed themselves.

None of the figures here are measurements. The two-billion-row join and the three-hundred-row delete are illustrations chosen to show the shape ^[1][4], and the shape transfers only if your deletes are scattered through files rather than clustered into whole files you could simply drop, and if your join keys are high-cardinality enough that the broadcast decision actually turns on the distinct count. The author works at Dremio, whose query engine consumes Puffin statistics, and says so in the piece while claiming nothing vendor-specific ^[18]. Read the ledger accordingly. An extensible container currently holding two blob types is mostly a bet on the third ^[16].