The cheapest LLM eval starts with one question: which mistake cannot be undone?

Reversibility sorts failures; accuracy only counts them. The tiers in this scheme put an over-ask below a confirmation that happened to be right this time, and both of those below anything already sent, deleted or charged ^[4]. A correct-over-total score cannot hold that ordering, because it weights every item identically by construction ^[20]. That is the practical case against borrowing someone else's benchmark: it hands you a rate without a direction.

One tier of the four is yours to write, which leaves three quarters of the rubric as copy work ^[19]. The expensive part is the sentence above it, and the examples in the post all resolve to the same shape, an act that has already left your system: an email you cannot unsend, a "decided" that other people start working from, an invented number now travelling upward in a report, an overwrite with no backup ^[3]. Whatever plays that role in your process is what the top tier means ^[2].

There is an asymmetry built into the grading. A wrong confirmation is worse than no confirmation ^[6], and over-asking costs only time ^[4]. A model that asks for confirmation on everything therefore never lands in either expensive tier. What keeps that from reading as a good result is the trap inventory, in particular the plausible non-targets, such as a spec question that contains a product name but is not an order ^[8], plus the handful of well-behaved cases the post says to put at the end ^[11].

Then the part most eval writeups leave out. Set the author's own defect count against the size of his finished exam and roughly one item in six carried a bug on the human side of the harness ^[18]. His mitigation is a flag recorded before the model ever runs: does the reference data alone pin this answer down to exactly one ^[15]. For "tape 60, 2 boxes" the catalog has a 60mm, so the data pins it and the key wins the argument ^[16]. For "clear tape, 2 boxes" no width is given, the data pins nothing, and "needs confirmation" was the right answer from the start ^[16].

The direction of suspicion is the load-bearing rule here. Re-read the data before docking the model points, but distrust yourself first if the fix turns a "needs confirmation" into a confident verdict ^[14][13], because that edit quietly promotes an irreversible action to a pass ^[4]. Sizing follows the same logic: one question per accident, each built to cause exactly that accident, so an inquiry read as an order becomes goods nobody wanted, a cancellation read as a new order ships twice, a misread unit ships fifty times the quantity ^[17]. If the system remembers anything between turns, the class where fresh information has to beat the remembered value stops being optional ^[9]. A day of that work is cheaper than a benchmark that scores the wrong thing well.