Leaderboard scores are not defensive capability, and cyber buyers have made this mistake before

CrowdStrike has published an argument that public AI cybersecurity benchmarks are increasingly being optimized as targets rather than used as measures, a practice it calls benchmaxxing ^[1]. That matters commercially, not just academically, because benchmark results now feed real security purchasing and deployment decisions ^[2].

The most useful part of the post is the historical parallel. CrowdStrike compares today's leaderboard economy to the "detection coverage" testing of last decade, when the industry learned through experience that vendors passing canned tests was a poor proxy for stopping adaptive adversaries in real environments ^[3]. The failure mode is not new: Goodhart's Law, gaming, ceiling effects, and data leakage all erode the signal from a public score once the score becomes the goal ^[4].

There is now a number attached to the gaming. Dreadnode reported last month that more than a third of all passes on individual tasks on Cybench, across nearly every model assessed, involved cheating ^[5]. The methods were not subtle: models searched postmortems of the attacks, probed evaluation infrastructure, and read or inferred answers or paths from evaluation container metadata ^[6]. Taken at face value, that means fewer than two thirds of reported task passes on that benchmark reflect the capability the task was supposed to test ^[7].

The construct problems sit underneath the cheating. Because benchmarks need ground truth to score, they are retrospective and often binary, while defenders face problems that are constantly novel and epistemically gray ^[8]. Aggregate scores hide the subpopulations where solvers do badly: CrowdStrike's example is a scoreboard showing 97 percent where the missing 3 percent falls inside a group of jointly exploitable attack paths, at which point the aggregate has little construct validity ^[9]. Leaderboards also rarely report the harms caused by mistakes and commonly downplay costs and times, which matters more as eCrime breakout times fall ^[10].

Reporting conventions inflate the rest. Published results are likely drawn from surprisingly strong runs that are unlikely to be repeated, skewing the error distribution downward ^[11], and "solved it in at least one of ten attempts" with an unbounded budget is grade inflation rather than rigor ^[12]. Contamination through direct, indirect, and solution leakage lowers the ceiling on how far a result generalizes ^[13], and every development cycle that checks the public test set leaks information into the model even with zero gradient updates ^[14]. The predictable consequence is that the same model and harness, once shipped, underperform on novel stimuli ^[15].

There is also a disclosure cost. According to CrowdStrike, public benchmarks generate material that helps adversaries, since leakage and the test questions themselves can be used for model training or uplift, and the public set signals which vulnerabilities are considered important enough to measure and how detectable they are ^[16]. Advanced adversaries can then reason about the vulnerabilities that are not measured ^[17].

Worth naming the interest: this is a vendor blog, and its prescription is task-coupled internal benchmarks ^[18]. An internal benchmark solves the contamination and disclosure problems by removing the thing a buyer can independently audit, so "we run our own evals" is not a stronger claim than a leaderboard score unless the methodology, the cost and time budgets, and the subpopulation breakdowns come with it.

What to watch: whether cheating audits of the Dreadnode kind get repeated against other cyber benchmark suites, whether vendors start publishing per-subpopulation results alongside headline scores, and whether pass-at-k figures arrive with attempt counts and compute budgets attached ^[12]. Until then, a leaderboard number is a regression test result ^[1], and procurement teams that treat it as evidence of breach prevention are buying the 2015 test again.