Per question, the fit estimates two things: how difficult the item is, and how sharply it separates models that are strong or weak on each latent capability. Per model, it estimates strength on those same capabilities [8]. A benchmark average is a sum over items that keeps neither. Every question gets weight 1, whether it discriminates or not, and whichever dimension it loads on.
The part that makes this more than a relabelling exercise is that the dimensions were not supplied. The authors did not tell the model which benchmark was measuring what, and the fit recovered two dominant dimensions on its own [11]. Reruns from scratch produced the same two [12]. That is the check I care about, because a two-factor solution you specified in advance will always find two factors.
It is also the limit. The corpus was six reasoning benchmarks and ten from the Olmo 3 safety suite [10]. Two dominant dimensions is what is identifiable from a corpus built out of two families. Add a code or multilingual family and the honest expectation is a third dimension, with the per-item loadings moving underneath it. Ai2's stability result rules out one random seed as the explanation for the two-factor fit, though it stops short of showing that the basis is complete.
The WMDP sign is the case that should change a scorecard. The benchmark counts refusing or failing to produce dangerous dual-use knowledge as the desired answer [16], so the metric is partly an ignorance test, and it is one that appears in safety tables. Read as a safety composite, an average that includes WMDP subtracts some fraction of reasoning ability from the total. Read as a capability probe, the same score means something coherent. The number does not tell you which reading its publisher intended.
Now the adoption bill. The parameters come from 100 models scored on more than 34,000 questions [9], which is at least 3.4 million model-question observations [18]. Discrimination is defined against that population of models. For the published loadings to transfer to your selection, two things have to hold: your candidate has to behave like something in that pool, and you have to be scoring the same items. If you are evaluating one fine-tune on your own prompt set, you have no discrimination estimates at all, and getting them means running many models over every prompt, not one.
So the practical version of this is narrower than "audit per prompt". Where Ai2 has published loadings for a benchmark you already run, stop reporting the composite and report the subsets, because BBQ aligning with reasoning [14] and the two halves of WildJailbreak [5] mean the composite is an average of different measurements. Where it has not, you are back to writing your own prompt-level splits by hand and defending them with an argument rather than a fit.
In my context, picking a model where over-refusal costs revenue and harmful compliance costs more, those two WildJailbreak halves are two separate purchases [4]. The number I want printed next to a safety average is the discrimination of its items on the safety dimension.
