Look at the delta column before the model column. Every nonzero mean delta in both tables is a multiple of 0.025 [3][4], which puts the scoring unit at one fortieth [13]. Mistral 24B's three positive iterations each moved exactly one of those units [4][5]. A paired comparison that shifts one unit does not clear p < 0.05 on a suite that coarse, and none of them came close: the lowest p-value across the ten reported iterations was 0.46, roughly nine times the gate [14].
The suite size itself is not recoverable from the post, and it is the first number I would want. Sixty percent is a clean count on 40 items. Sixty-four percent is 25.6 items, and on the 10-task set used for the earlier 9B and gpt-4o-mini spot checks it would be 6.4 [11][15]. So the accuracy column and the delta column are not counts over the same set, and the denominator is what decides the smallest effect the gate can ever see.
Now the loop. Each iteration the analyzer reads the failures, emits one proposal, the harness A/B tests it, the gate rejects it, and the next round is handed the same failure list [8]. Nothing in that sequence records which class of edit just died. So the analyzer re-derives the same diagnosis from the same evidence and rephrases it: add "active" to security breach, add "only if critical to business continuity", add "requires immediate intervention" [7]. That is one candidate with a thesaurus attached.
The interesting edit is the one nobody wrote. Multi-label tasks return single labels, and the author's own estimate is that forcing comma-separated pair output would move three tasks straight away [10]. If a scoring unit is a task, that is a delta of 0.075, three times the largest any model produced [16]. It sits inside a failure family the analyzer had already named in its own report [9].
For this result to transfer to your loop, three conditions have to hold: your promotion gate is a significance test over a suite small enough that a one-item gain fails it, your analyzer sees only the previous round's failures, and you sample one candidate per round [8]. Widen the suite and a single-task edit becomes provable. Sample k candidates with a ban on previously rejected classes and the local optimum stops being an absorbing state. Neither change needs more parameters, which is about what the 1B-to-30B sweep showed: the 1B could barely function as an analyzer, and above that floor the extra weight bought latency rather than better proposals [2].
This is one author's project across two releases [1][9]. Take the mechanism and leave the effect size. Single-candidate hill climbing with no memory of rejected moves gets stuck in a local neighbourhood, which was true long before any of these checkpoints existed [12]; what the 4,150 calls contribute is a price for confirming it the expensive way [1].
