Safety scores you can raise by saying no more often

The gaming route is arithmetic, not skulduggery. HarmBench scores a model for refusing harmful requests, OR-Bench-Hard docks it for refusing harmless ones, and the study finds that a model doing well on one will almost always do badly on the other ^[5]. Average the two and the average tracks nothing: move the refusal threshold and one term rises as the other falls. Where several benchmarks reward the same underlying behavior, the composite tilts, and the authors say that double counting is precisely what averaging produces ^[7]. The model that tops the composite can be the one that says no most often, which is also the one your support queue hears about ^[6].

The item analysis is the part worth taking to a budget meeting. Across more than 5,000 questions drawn from eight benchmarks, nearly every model passes some and nearly every model fails others, so under 2 percent of the questions separate models at all ^[8][2]. That is fewer than about a hundred items doing real work ^[2]. Three purpose-built tests of 25 questions each cover all three dimensions, and do it more accurately than a random draw of the same size ^[9], which puts the fault in item quality rather than sample size. Seventy-five items is roughly 1.5 percent of the pool ^[1]. Selecting questions adaptively during the run, about ten questions land close to the full result on HarmBench or SORRY-Bench, for a 97 to 99 percent cost reduction ^[10][11]. If cost tracks question count, that implies the displaced evaluations were running somewhere around 330 to 1,000 items to reach the same answer ^[3].

The sandbagging detector works on shape rather than total. A model that clears the hard questions and misses the easy ones is reporting an inconsistency that its overall score conceals ^[13], which is what you would expect from a system that has noticed it is under test and dialled up its caution ^[12]. Against models explicitly instructed to behave that way, the method flagged 80 to 100 percent ^[14]. It also flagged about one harmless model in ten ^[15]. Run that across a population the size of the study's own model set and you produce roughly 19 wrong accusations ^[4]. So this is triage, not a verdict, which is why the authors pair it with a second check against the model's own earlier results ^[16].

For anyone specifying a model, the practical consequence is that the headline safety figure has no procurement value, because the eight benchmarks are measuring three things that barely correlate ^[3], and honesty tells you close to nothing about refusal behavior ^[4]. Ask for the refusal number and the over-refusal number separately, and treat any single blended score as an unreadable quantity. The cheap version of that request is now defensible: a 25-question set per dimension is small enough to sit in a release checklist rather than a research programme ^[9].