Leaderboards as a procurement trap: when the test rig outranks the model

Rank is a property of a configuration. The post stacks three sources of variance between a published number and a deployed system: harness, data, and elicitation ^[16]. A rank that flips when the rig changes ^[15] is not a property of the thing a buyer signs for.

Hold the harness constant and two failure modes remain. Ground truth is a claim rather than a fact, and it can be wrong, contaminated or outdated, so an evaluation built on a bad label returns precisely calibrated wrong conclusions ^[1]. Elicitation cuts the other way: a score is a lower bound on capability and never an upper bound ^[2], which means the system ranked second may be the better one under a different scaffold. Neither error is visible from outside the leaderboard, and neither is corrected by waiting for the next release.

The arithmetic worth putting in a procurement file comes from the second pillar. Red Hat cites Lee's five evaluation surfaces and says most teams cover only the first ^[17], which leaves four of five, 80 percent of the inspectable behaviour, unexamined ^[19] while the buying decision rests on a number summarising the smallest slice. The reason that matters operationally is attribution: terminal state cannot tie a failure to a particular action, so the per-step diff is the evidence and the final state only the verdict ^[6].

What replaces the leaderboard is not cheaper, and the post is candid about it. Progress is gated on verifiers because generation now outruns evaluation ^[3], and delegating the judging to a model imports every defect of the model under test, since LLM-generated evaluators inherit the problems of the LLMs they evaluate ^[4]. Even the plumbing bites: without clean trial isolation, what gets measured is an infrastructure artifact rather than agent capability ^[8]. That is the real cost of task-level correctness. Somebody has to write down what good means for the specific task, then build a rig that cannot be gamed by leftover state, and neither job produces a comparable score for a slide.

The limit of this argument is where the source itself stops. Our copy breaks off inside the fourth pillar, which announces four distinct success metrics without listing them, and the fifth pillar never arrives ^[11][12]. So a post whose thesis is that benchmark scores rarely translate into business value ^[13] ends before it says which numbers should stand in their place. That is not a small gap. The whole case against leaderboards is that a single headline metric hides the stack that produced it, and the replacement proposed so far is a set of measurement disciplines rather than a metric a finance function can hold a vendor to. Buyers who accept the diagnosis are left doing the definition work themselves, on their own tasks, with their own harness, and comparing agents on evidence nobody else can publish.