Two canaries with known outcomes tell you whether an agent eval is scoring the model or the harness

A canary everyone expects to pass, which then fails, is reporting that the environment moved, and the pass rate computed afterwards describes that movement rather than the agent ^[13]. That is the direction most likely to slide by, because a lower score reads as a harder task set. The sandbox image digest belongs in the lockfile for the same reason: package churn inside an image is an unlisted independent variable ^[18]. Clock source has the same property for wall-time, since measuring on a busy laptop mixes agent latency with thermal throttling and browser updates ^[19].

Grader isolation is, in the post's account, the control a small team assembling a weekend harness skips first ^[14]. The rule has two halves: the agent does not see the hidden tests, and the grader does not see chain of thought unless the protocol says it should ^[15]. Break the second half and the artifact is specific, because shared context is how a model appears to understand a constraint that never appeared in the user-visible prompt ^[16].

Task families are supposed to be gradable without watching the agent think out loud, each with a public cousin for debugging and a hidden cousin held back until the prompt hash is locked ^[26]. Duplicate statements of the same bug collapse into one holdout item, so the percentage does not count a single skill twice ^[25]. Difficulty labels stay metadata, since they are operator judgments and not a second score competing with pass rate ^[27]. That bookkeeping is doing real work: prompt text, tool allowlists, sandbox images and grader prompts often move the score more than the model does ^[2]. The registry exists to lock those six fields down before a scored run, not to add YAML to a README ^[31].

Two further columns decide whether a ranking exists at all. Variance across seeds goes in the same row, because a gap inside the noise band is not a ranking ^[21]. Cost goes in beside it, since two agents that pass the same hidden tests at very different spend are not interchangeable ^[22]. A schema that refuses incomplete score files covers all of this; no dashboard is needed ^[24].

The post's scope stops short of an audit. It presents a methodology argument and reports no measured drift and no named vendor's leaderboard figures ^[30]. So the case that published agent percentages are unverifiable rests on the missing fields, not on a demonstrated failure of a specific claim. The transfer test it yields is usable anyway. Another team's pass rate says something about your repository only if their frozen temperature and capped tool-output budget resemble your runtime ^[17], and only if their image digest is pinned well enough that a rebuild does not silently change the variable ^[18]. Where those are unstated, a gap between two agents is not attributable to either agent.