Catching a model-swap regression takes twenty labeled documents run twenty times each

Apply the post's own comparison rule to its own table and the second row gets close. Model A's notice_period_days interval runs 84.1 to 92.9 percent; model B's runs 74.0 to 84.4 ^[15]. Those overlap between 84.1 and 84.4, a span of 0.3 points ^[1]. The row is labeled "separated, real regression" ^[15]. The point estimates sit 9.3 points apart and the intervals still touch ^[2]. Under the rule as written, the strongest example in the piece is a third of a point from inconclusive.

The suite is twenty labeled documents run twenty times each ^[10][11], which comes to 400 extraction calls per model ^[4]. Comparing an incumbent against its replacement doubles that to 800 ^[4]. Each run scores eight fields ^[7], so one model evaluation produces 3,200 field-level hits and misses ^[5]. Labeling comes first, on documents like the 30-page service agreement where the notice period is in clause 12 and an amendment on page 27 overrides it ^[3], with ground truth agreed once and then left alone ^[10].

Those 400 calls are not 400 independent samples. Twenty trials against one document are correlated, because a document the model reliably reads wrong contributes twenty misses rather than one. The published widths fit that. A normal-approximation 95 percent interval on 400 independent trials at an 88.5 percent hit rate is about plus or minus 3.1 points ^[6]; the table shows plus or minus 4.4 ^[12], which is the width you get near 200 samples ^[7]. The table is illustrative, so the figures may simply be hand-written. In my view the interval should be resampled over documents, not over calls.

Field-level ranking is a claim about one corpus. liability_cap lands lowest at 71.0 percent because in these contracts the value is buried in carve-outs and cross-references ^[12][13]. For that ordering to hold on your documents, your weak field would have to be weak for the same structural reason. The output format transfers: a hit rate per field with an interval around it.

A single run cannot stand in for that, because even at temperature=0 identical outputs are not guaranteed, and batching, hardware and silent server-side changes all move the result ^[5]. "A single passing run is one sample from a distribution you have not measured," the author wrote ^[6]. The reason the swap goes unnoticed is that a wrong value is shaped like a right one: 60 days is plausible for a notice period, and so is 30, which is what the superseded clause says ^[4].

The first harness most people write asserts one value per field on one document ^[8]. The post calls that a smoke test that tells you the pipeline is wired up, says nothing about accuracy, and flakes until somebody marks it @pytest.mark.skip ^[9]. The author wrote that the whole thing can be scripted in an afternoon, and that he has written versions of it more than once ^[17].