Test the rule change, not the model: a cheap A/B harness for agent standards

Run the arithmetic on the recipe before building it. Two arms of ten to thirty tasks is twenty to sixty agent runs per rule change ^[4], which is cheap enough to do between other work. It is also small. Thirty tasks scored pass or fail resolve nothing finer than one task, about three and a third percentage points ^[1]. The guide's own cautionary example, a standard that lifts quality by 3% while doubling token use ^[10], sits under that floor: 3% of thirty tasks is nine tenths of one task ^[2]. The doubled token count you will see, because it is counted rather than judged. The quality gain you will not, and if you adjust the rubric until you do, you have rebuilt the evidence-free rule change the harness was meant to replace ^[2].

So point it at delivery instead. The failure the guide calls unreliable instruction delivery is the one that shows up at this sample size: the wrong instruction file loads, the rule sits too deep in context to matter, a stale example gets over-followed, or no skill is selected at all ^[4]. A migration safety skill that never loads because the ticket said "fix signup bug" is not a 3% effect ^[6]. It is a count, and counts survive twenty runs in a way rubric averages do not, which is why the guide tells you to measure selection reliability and not only output quality ^[13].

Conflicts are the second thing worth a fixture. When one file says prefer fast minimal changes, another says always add complete tests, and a third says avoid touching test snapshots, precedence is being decided at runtime, per task, by the model, and nobody on the team has written down which rule wins ^[8]. A conflict case in the task set turns that into a reviewable artefact, which is the actual argument for treating standards like code: versioned, reviewed, tested, rolled out ^[14].

The cost side needs a verdict per workflow, not one global answer. Take the guide's own tradeoff at face value: 3% more quality for twice the tokens is roughly 94% more spend per acceptable output ^[3]. The guide's position is that this can be fine for high-risk work and probably is not fine for every background automation ^[11]. That means the same variant can pass and fail in the same organisation, and the harness has to record which population it was scored against.

What makes any of this affordable is the control on the runner: same agent, same model family, same tool access ^[12]. Nothing about model capability is under test, so no ground truth about the model is required. Your prose is under test, which is the question teams actually have when they add a rule ^[3]. Worth noting that the source stops mid-sentence on the scoring rubric, so the cheapest part of this design is specified and the expensive part is not.