A prompt A/B on six inputs is a coin flip until you have measured the noise floor

The wobble has a mechanism, and it is not the temperature slider. A low setting is not a zero one. Batching and sampling move the output, and so does the order inputs happen to arrive in, and at small sample sizes that movement is larger than most prompt edits ^[7]. Which is why the first thing worth measuring is the harness rather than the change: run one prompt twice, compare it against itself, and treat the result as the bar every later comparison has to clear ^[8].

The sample size problem shows up immediately after. Beating a noise floor takes tens of items, and the author of the dev.to post is blunt about where hand review gives out: six comparisons fit in a head, thirty do not, and by item twenty the standard has drifted, because tired reviewers get generous ^[9]. That is roughly a fivefold gap between the sample the arithmetic wants and the sample a person can actually hold to one standard ^[17]. The reading method breaks before the statistics do.

So the checks have to be things something can count. His five, for a summarizer: contains a digit, headline word overlap above about 70 percent, a filler opener such as "The article describes", mixed scripts inside a single word, and not one character of the target script ^[10]. None of them is a judgment of quality, and he does not pretend otherwise. The stated rule is that a metric has to capture a failure you can name, and quality is whatever is left after the named failures stop happening ^[11]. That puts the real prerequisite ahead of any harness: a written list of the ways the output has already embarrassed you in production ^[12].

Counting discipline carries as much weight as the checks. Percentages across two fresh runs reproduce the same coin flip at a larger scale, so he seeds the query instead of re-sampling, gives both versions byte-identical inputs, and counts per item rather than in aggregate ^[13]. When the aggregate and the paired count point the same way, that is a signal. When they disagree, the finding is no difference found ^[14].

The result that pays for all of this is one a side-by-side read would have got backwards. He replaced a wordy instruction, which asked for exactly one detail the headline lacked and listed banned phrasings plus a written-out bad example, with something short: start immediately with the fact ^[15]. The terse version produced fewer concrete details and close to twice as many summaries that just restated the headline ^[16]. Note how that verdict became visible at all. Headline overlap was already one of the five counters, so the failure arrived with a number attached ^[19]. Read by eye, a short confident restatement of a headline looks a lot like a good summary of an article.