Slicing FrontierMath by category leaves about 13 problems behind each error bar

A CLT-based interval is an asymptotic approximation, and its accuracy depends on how many datapoints you actually have. The paper's position is that thousands of examples is enough for that approximation and fewer than a few hundred is not ^[2]. Below that, the failure has a direction: the bars come out too small ^[3].

The headline size of a benchmark is not the sample size a decision rests on. FrontierMath holds around 300 problems across 23 categories, which averages about 13 problems per category ^[12][17]. SWE Bench Verified is 500 samples across 4 difficulty levels, so 125 per level ^[14][18]. CUAD's 510 labelled documents span 25 contract types, roughly 20 per type ^[11][19]. AIME, as the paper counts it, is 15 problems, and MLE Bench is 75 Kaggle competitions ^[13][15]. Some FrontierMath categories carry fewer than three samples ^[12]. Two samples will still produce a mean, and a CLT interval will still be drawn around it.

For a published delta between two models to transfer to your deployment, two things have to hold at once. The eval set has to stand in for your workload, and it has to be large enough for the interval quoted beside it. The paper argues those requirements pull apart. MMLU, GSM8K and Big Bench run hundreds to thousands of examples, and the paper describes them as largely saturated tasks, such as high school science questions, that do not accurately represent the tasks LLMs are used for in practice ^[8][9]. The targeted benchmarks built by industry practitioners and by researchers probing frontier capabilities carry very high quality labels and often only tens to hundreds of examples per task ^[10].

The reason those sets are small is annotation cost. CUAD required law student annotation with review by experienced attorneys, at an estimated $2 million for 510 documents, which works out to roughly $3,900 per document ^[11][20]. FrontierMath was developed in collaboration with over 60 expert mathematicians ^[12].

The abstract puts the failure in words, not in a coverage figure: CLT-based methods "perform very poorly, usually dramatically underestimating uncertainty (i.e. producing error bars that are too small)" ^[21]. How far a nominal 95% interval actually falls short at 15 problems is a question for the paper's experiments. What the supplied text supports is a threshold, a direction, and a list of benchmark sizes that sit under the threshold ^[2][12][13]. Those are the evals that inform model selection and deployment decisions ^[7].