Pull the outlier and the headline moves. If each of the six setups faced the same 96 absent fields [5], the pooled 17% [8] is a straight average, and removing the 40% row [7] leaves (17 x 6 - 40) / 5, about 12.4% [1]. Velrim's own 11% [6] sits inside that band. So 17% is a property of a table with Mistral in it. A stack that does not include Mistral should carry a prior nearer 12%.
The finding also rests on a small ledger. Six systems against 96 absent fields is 576 chances, and 17% of that is roughly 98 invented values [3]. Per system, 11% is about eleven fields and 40% is about thirty-eight [2]. Velrim says as much in its own framing: where the uncertainty brackets overlap, the test cannot separate two systems [24], and per document type, gaps under about 4 to 10 points are invisible at this corpus size [17]. Read the table as one outlier and five rows that tie.
One thing does not reconcile. The answer keys are described as holding 96 absent fields [5], while the team also says it hand-checked all 142 absent-field labels against the actual pages before any competitor system ran [23]. That is a 46-label gap [5] the post does not explain, so count the keys in the repo before you quote a rate. The sequencing is right either way: labels frozen first, competitors second.
For the rate to mean anything in your pipeline, your documents have to look like those 124 [4] and your schema has to ask for absent fields at a similar frequency. The measurement is per absent field, so your exposure across all returned fields is the absent share multiplied by the fabrication rate. A schema of mostly required fields carries very little of it. An optional-heavy schema carries nearly all of it. Velrim's weakest accuracy row is TV ad contracts at 0.59 out of 1 [16], which tells you which document families are hard, not what your own score will be.
The decoding ablation is the part I wanted more of. Constrained and structured runs force the output to match the schema as the model writes it, while free-decode runs write freely and get checked afterwards, and on this corpus that choice is worth under 2 points for OpenAI's model and nothing detectable for Gemini [18]. That comparison is reported on accuracy.
The pre-registration is the craft worth copying. Section 10 of the committed plan states that the expected accuracy outcome between Velrim and the bare model underneath it is a statistical tie, signs that tie as publishable before any money is spent, and adds that a design needing Velrim to win the F1 column is a failed design [12]. The run came in two points behind bare Gemini, a statistical tie [13]. The post also reports a DIY setup on OpenAI's small model beating Velrim by about four points [14], and bare Gemini beating it clearly on US registration filings [15]. A vendor paying for a benchmark that ends with "if a wrong extracted field costs you nothing, the bare model is a much better purchase" [25] is not the usual genre.
What is reusable here is the answer key with verified absences in it, plus the balancing column that stops refusal from being a winning strategy [22]. Without both, an extraction suite scores the fields that exist and stays quiet about the ones that do not [3].
