Per-property pass bars expose failures that a 92 percent eval average hides

OpenAI's sentence, as the post quotes it, starts with "also": "We also didn't have specific deployment evaluations tracking sycophancy." ^[2] The word implies at least one other gap. The post quotes only that line, so it does not show the rest of OpenAI's account. The author's summary, "They weren't checking for it, so it got through," fits the gap the line names. ^[21]

The remedy the author proposes is old: decide what good looks like before release, check for it continuously, and make failure visible. ^[3] Kent Beck's Test-Driven Development: By Example came out in 2002, and the post notes Beck has described TDD as something he rediscovered. ^[5] The part that breaks is the assertion. assertEquals(expected, actual) assumes one input gives one output. A sampled model can answer the same question two ways with nothing changed but the draw. ^[4] The author replaces the assertion with a named property and a minimum success rate. ^[6]

The sample run covers 24 scenarios with three repeats each, 72 runs. ^[6][12] The repeats measure sampling variance on those 24 prompts and add no new cases. For this suite's verdict to transfer to another agent, its scenarios would have to resemble that agent's traffic.

Each property carries its own bar. Safety and gate-outcome sit at 1.00 and scored 1.000. ^[7] The author's reason is that "mostly safe" isn't a meaningful standard. ^[9] At that bar, one bad safety run drops the rate to 71 of 72, about 0.986, and fails the suite. ^[17] Intent accuracy scored 0.875 against 0.90, or 63 correct of 72. ^[8][16] Passing needs 65.

The author puts the rolled-up score at 0.919. ^[10] I got the same figure by pooling every judgment, weighted by each property's N: 204 passes out of 222. ^[13] A plain mean of the five rates gives 0.908. ^[14] Both clear 0.90, the bar two of the properties missed. The answered rate of 0.667 over 27 works out to about 18 answered and 9 left unanswered. ^[15]

The N column follows applicability. Groundedness was scored over 18 runs because it only applies when the agent retrieved something. ^[11] "A denominator should describe reality, not improve the metric," the author wrote. ^[20] Per-property bars and honest denominators are careful engineering. In this run the suite failed while both averages sat above 0.90. ^[8][13][14]

In my view, the check OpenAI named would fit this format as one more row, with its own bar and a denominator limited to prompts where agreeing with the user is the risk. The author wrote: "The genuinely difficult part of evals isn't that models are mysterious. It's choosing the right test." ^[19]