Prior cybersecurity benchmarks graded vulnerability reproduction, patch generation and capture-the-flag puzzles, and frontier models already score well on many of them [11]. Those tasks mostly reward reading source. ExploitGym grades runtime behaviour instead: heap metadata, stack frames, virtual memory mappings, instruction-level control flow, register state, and inputs that satisfy tight constraints [10]. The task is escalation, from an input that crashes something to unauthorized file access or code execution [8], sustained over a long horizon while the process state moves under the agent [9]. A model that writes a correct patch has demonstrated nothing about whether it can hold a heap in a known shape across a hundred allocations.
Then the arithmetic. 157 working exploits out of 898 instances is 17.5 percent [13]. GPT-5.5's 120 is 13.4 percent [14], which puts Claude Mythos Preview ahead by about 1.3 times on raw count [15]. Those percentages only mean what they look like under conditions the abstract does not state. Both configurations would have to have been run against the full 898. An instance would have to be a distinct vulnerability rather than a vulnerability crossed with a protection setting, and since the authors vary the protections applied to each instance [5], one bug can plausibly appear several times, which would make the per-bug rate higher than the per-instance rate. And the reported leaders are described as configurations, not models [4], so the number grades a scaffold as much as a set of weights. Swap the harness and the count moves.
The bigger read-across problem is the test condition. Main experiments ran under trusted-access programs with safeguards disabled, explicitly to measure the capability boundary [7]. That answers the question the paper asks, but it describes what the weights can do rather than what a stranger with an API key gets out of the shipped product, which is not the number a defender needs for a threat model. Those are two different measurements and the abstract supplies one of them.
Same gap on mitigations. The paper reports that models retain non-trivial success rates with widely used defenses enabled, without a figure attached in the abstract [6]. Non-trivial is the least falsifiable number in the paper.
What survives all that caveating is still useful, and it lands on triage. Exploitation is what converts a severity argument into a demonstration, and the paper lists exactly that defensive use: assessing severity, prioritising patches, validating mitigations, alongside the acknowledgement that it lowers the expertise needed for offense [12]. If your process defers fixes on the strength of "crashes, probably not exploitable", this is the yardstick that eventually prices that habit.
The design choice worth copying is the one that gets least attention in the abstract. Varying the protections per instance and shipping every configuration as a reproducible container [3][5] turns a leaderboard question into a measurement of how much each mitigation costs an agent that can already exploit. That is a benchmark you can re-run against your own build flags instead of taking on faith.
