Identical 0.95 confidence on right and wrong answers killed Scribe's threshold gate

A threshold gate needs separation between the right answers and the wrong ones. The confidence scores did not separate correct values from wrong ones: median composite confidence was 0.95 for correct values and 0.95 for wrong ones, and that killed the plan to tune the accept/flag gate on self-reported confidence ^[2]. A median is the middle of a distribution, and with a ceiling at 1.0 there is little room above 0.95 to put a cut anywhere useful. The log reports handwritten proper names misread with total confidence ^[3].

What replaced it is a verify pass. Every field the pipeline would auto-accept is re-read by a different model, and disagreement flags the field for a human ^[12]. The signal is now two reads that can differ. It is not free, and the bench says so: mean wall-clock seconds per field includes verify where present ^[14].

Across the nine configurations, accuracy on auto-accepted fields went from 75% to 96%, while exact accuracy over all fields went from 75% to 81% ^[17]. The gain on what skips review is 21 points; the gain on reading is 6 ^[22]. The author reports the models did not learn to read much better and the pipeline learned when they were wrong ^[17].

The reading numbers came from the rest of the pipeline. Two configurations, v4 and v8, changed only the verifier, and their reading-accuracy columns track their primary's row ^[18]. The other stages moved exact accuracy: a grounding call localizes every field on the page at once, tables, dates, enums and checkbox groups are read from margin-padded crops while scalars and prose come from the whole page, and a validator enforces schema and types before anything is accepted ^[12].

For the 0.95 collapse to transfer, the fields have to look like these fields. Of 221 hand-curated gold values, 51 are legitimately null because the entry is absent, illegible, or non-numeric where a number is required ^[9]. That leaves 170 values with something actually written on the page ^[23]. Three of the 23 pages are synthetic, built to carry specific traps ^[8]: an adult's age written as "Ad" in an integer field, an infant's age given in months, four date formats, a patient identified only by hospital number and bed, and a tick-box grid with nothing ticked ^[10]. "A model that guesses politely is worse than a model that abstains, because a guessed value in a clinical record looks exactly like a real one," the log says ^[11].

None of this ranks models, and the author says so first. The corpus is the public African Medical Records dataset under CC-BY-4.0, pinned at revision fa27d29 ^[19]. Public data may overlap the training corpora of every model tested, and one of the models tested was released after the dataset was published, so the numbers compare configurations under identical conditions and measure the pipeline, not the models' reading ability ^[20]. Those identical conditions are one host: a MacBook Pro with an Apple M5 Max and 128 GB of unified memory, models served OpenAI-compatible through mlx-vlm with grammar-constrained JSON decoding ^[7].

`scribe run` holds a process-wide socket guard that blocks any connection off loopback ^[5]. The offline claim is testable from the process table. Every extracted field also carries provenance back to a region of the page, and anything uncertain is routed to a human instead of into the record ^[4].