Leadership1 publisher2 min readPublished
Whoever writes the eval decides what an AI program is allowed to get better at
An essay at highperformanceailab.com argues that the measure a team picks defines what counts as progress, and that a perfectly calculated score can be satisfied by work nobody did. Its case rests on analogy.
The Board Room · Leadership desk
What happened
- An essay published on highperformanceailab.com argues that an eval is a theory of success made operational, not a test applied at the end of the work.
- It notes that AI makes such verdicts easy to produce, since a model swap or a revised instruction can be tried in hours and an automated loop can run a hundred variations overnight.
- Its worked example is a support operation that measures resolution by ticket closure, tunes an AI assistant against that target, and becomes good at closing tickets whose problems remain unsolved.
- The piece ends by introducing ProofPack, described as an independent verification and evidence layer for evals in which known-good cases test whether the evaluator is merely refusing.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
- constraint A program can only improve at what its eval observes, so a definition nobody revisits sets the ceiling however many cases are added beneath it.
- decision The scheduling choice is when the eval itself gets reviewed relative to the loop optimising against it, because after pay and releases depend on the number that review becomes a budget negotiation.
- exposure Anyone signing off on a score inherits the assumption that the evaluator was checked; if it was not, the signature certifies the calculation alone.
- precedent If verification of the evaluator becomes something a firm buys, the renewal conversation moves from how the model scored to who audited the scorer.
Two changes can raise the same score for opposite reasons. One actually improves the system. The other learns what the grader rewards, and on the dashboard the two look alike [4]. The essay's image for the gap is a smoke machine, which produces the smoke without the fire: "Nothing about the smoke is false. Only the conclusion is," it says [10]. The inference an operator actually makes runs from the work to the cases, from the cases to the evaluator, and from the evaluator to the number [11]. A system that satisfies the evaluator without completing the work breaks that link, and the score is still calculated perfectly [12].
If the eval defines what a process can recognize as improvement [2], it also fixes what the program cannot get better at, because behavior the eval never observes does not register. Volume does not correct that. "More test cases can produce more smoke. They do not establish that there is a fire," the essay says [13].
The essay's claim concerns what the number is permitted to do: it asks more than whether a system passed, extending to what the passing result proves, how that proof can be checked, and what the result is now allowed to change [5]. Once budgets, promotions and releases depend on the number, it argues, the abstraction begins remaking the reality it was meant to describe [15].
Four of the essay's questions are aimed at the evaluator itself: can it reject behavior known to be wrong, can it keep accepting behavior known to be honest, does an irrelevant change alter the verdict, and what happens at the edges of the cases somebody thought to include [14]. Writing them down costs an afternoon. Answering the third means re-running cases whose verdict nobody expected to move.
The support here is one essay on highperformanceailab.com [1]. Its evidence is analogy and a hypothetical support desk; the text does not report a score or a named program [2]. It says plainly that it is not an argument against evals; the argument is for keeping the map answerable to the journeys people actually take [17]. The site making the argument also publishes the verification layer that answers it [1].
What to watch
- Whether ProofPack publishes its known-good case sets and pass criteria where a buyer can inspect them.
- Whether any operator publishes a before-and-after of an eval it rewrote, with the score movement attributed to the changed definition.
- Whether AI service contracts begin naming the evaluator as an auditable deliverable alongside model output.