The gate's sensitivity is a property of the suite, not of the regression, because the score is passes divided by case count [5].
Run the arithmetic on the three cases the post ships [9]. One failure costs 0.333 of the score, more than six times the band [1]. With nineteen cases, one failure costs 0.0526 and still fails. With twenty, it costs exactly 0.05, which is not less than -0.05, so the diff merges green [2]. Every case added past the twentieth makes the gate less sensitive, even as it adds coverage. Single-case strictness at scale means comparing per case, or setting the band to zero.
The stored artifact cannot support the per-case version. Two runs at 0.90 are identical to the gate and need not share a single passing case [3]. Prompt edits that buy formatting compliance with refusal behaviour produce exactly that shape, and the author's own account of where regressions hide names formatting constraints and refusal behaviour together [11]. The failing ids and the first 160 characters of each bad output do get printed to the console [c6a]; they are simply not in the file the next run reads [7].
Graders are only as tight as their strings. grade_keyword lowercases the output and returns true if any keyword appears anywhere in it [3], and the shipped sentiment case looks for "negative" [9]. An output reading "this is not negative, it is neutral" passes [4]. A grader that agrees with you under negation is not actually checking the claim it's supposed to check. grade_word_count splits the entire response [4], so a polite lead-in spends words the summary needed [5].
Then the config line that sets the noise floor: temperature is 0.2 [c8b]. Sampling variation therefore lands in the delta alongside the prompt change, which means part of the 0.05 band is an unpriced flake allowance rather than headroom for real drift [6]. Pinning temperature to 0 costs generality and buys a delta that means what it says.
Cases run one model call each, in sequence [5], so a prompt diff pays N serial calls per CI run [7]. At three cases that is invisible. At three hundred it is CI wall-clock and a per-pull-request bill.
The craft here is in the smallness: the author keeps the harness to one file because a complicated eval framework is one more thing that can break silently [14], and the argument against relying on spot-checks is well made, since humans test happy paths and language models are stable there [11]. The author treats an eval suite as a required artifact of any prompt change [12], which is the schema-migration posture, and it is the right one. What each adopter has to set for themselves is the -0.05. It works for a short suite of near-deterministic calls; a larger suite or looser sampling needs a different number entirely.
