Build1 publisher3 min readPublished
Rerunning the same eval suite three times in ten minutes moved its score by two cases
A developer who built his own eval gate reran an unchanged 21-case suite three times in ten minutes and got three different scores, with the run files recording identical prompt checksums each time.
The Engineer · Build desk
What happened
- The three run files carry the same config hash and the same sha256 for both prompt files, and git records no commit touching prompts/ between the first run and the third.
- Brief's 21 suite cases are all items the tool had already shown him by August 25, 18 of them scored 4 or 5 and 3 below-threshold fillers added by a pad-to-five rule.
- He reverted two JUDGE.md edits on September 9, then moved scout to Sonnet 5 the same day and re-set the gate's thresholds on that model's run.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A prompt edit worth one or two cases cannot be told apart from the suite's own 9.5-point spread, so this gate can only adjudicate changes larger than a noise level it had never measured.
- decision Anyone setting a threshold now has to decide whether to spend model calls repeating the baseline config first, or accept that the promoted number inherits one run's luck.
- exposure Every scout result recorded before the Sonnet 5 switch is now held against a gate its model never faced, so the repo's stored history is not a comparison series.
- constraint Ground truth harvested from what a filter surfaced measures agreement with the filter, and no volume of extra runs will produce the missing labels.
Five samples per case with a majority vote means one sample has to move for a case verdict to flip. Seven of brief's 21 cases had at least one run where the five samples split [6], a third of the suite [3]. Two flips is the whole distance between the best and worst of the three runs, and on a 21-case suite one case is worth 4.8 points of accuracy [7], so the spread came to 9.5 points [1]. Precision moved with it, 10/15 down to 9/15 and back up, a spread of 6.7 points [4] [2].
Nothing in the configuration accounts for that. Each run file stores the config hash and the sha256 of both prompt files, and all three sets match [5]. The author, who built the gate his suites run through [2], wrote: "I built the gate, so I had every reason to believe the numbers" [7]. He came to the experiment by rereading Dan Luu's exercise 7, which shows a benchmark as published and asks what is wrong with it before explaining anything [8].
Scout is the second judge. It returns comment, upvote or skip from rules kept in a file called JUDGE.md [9]. On September 9 the author tested two of those rules against a reference of 17 labelled threads on Haiku 4.5 at five samples a case, Edit A adding a paragraph and Edit B adding a single line [10]. Both went back the same morning. He then moved the judge to Sonnet 5 and re-set the gate's thresholds on that model's run, and the Haiku reference the two edits had been measured against would not pass the new gate [11]. In his words, "the edits were tested against the reference of the moment, not against the baseline promoted afterwards" [12].
Brief's ground truth comes out of the product loop. It scores each item 1 to 5, shows him everything at 4 or 5 or the top five if fewer, and asks which ones interest him; that answer is the label [13]. The file held 446 items on September 6 [16], and the suite draws 21 of them, 4.7 percent [4]. The post asks which metric cannot be computed from that table "no matter how many runs I do" [19]. Recall is what the file forecloses: an item the loop never surfaced never got a label, so its false negatives are not in the data at all [6]. Precision reads 10/15, and the gate has been green for two weeks [15].
The 9.5 points are a measurement of 21 cases scored by Haiku 4.5 at five samples each. Two conditions have to hold before that spread means anything on another rig: the per-case verdict has to come from a sampled vote that one sample can flip, and enough cases have to sit near the split. Suite size sets what a flip is worth. On September 11 the author regenerated scout's suite from 144 threads labelled with what he had actually done on Reddit, up from 17 cases [17]. One case there moves accuracy by 0.7 points [7].
The finding cost three runs of an existing suite inside ten minutes [3].
What to watch
- Whether the regenerated 144-case scout suite gets repeat-run variance measured before new Sonnet 5 thresholds are fixed.
- Whether brief's ground-truth file starts carrying labels for items the judge filtered out, the only route to a computable recall.
- Whether the remaining exercises in the series report the same spread on the scout suite as on brief's.