SecurityNot yet confirmed elsewhere1 publisher2 min readPublished
Claude Opus 4.7 crash fixes failed maintainers' tests a third of the time in Northwestern study
Northwestern tests of AI agents on 112 real bugs found a third of Claude Opus 4.7's crash-stopping patches failed maintainers' tests. Many patching benchmarks grade only that crash, so their scores overstate how many bugs are fixed.
The Watch · Security desk
Newest Claude model passed maintainers' tests about 61% of the time but stopped the crash about 90%. The study used 112 real security bugs from 84 open-source projects. Build-and-test instructions lifted C/C++ success by 20 points. The team reviewed 13 benchmarks and datasets.
- 61% Passed maintainers' tests, claim 7 About; it stopped the crash about 90%
- 112 bugs Real security bugs tested, claim 1
- 84 projects Open-source projects covered, claim 5 C/C++, Go and Rust
- 20 points Gain from build-and-test prompt, claim 16 On the C and C++ bugs
- 13 reviewed Benchmarks and datasets reviewed, claim 15
| Figure | Value | Note | Claim |
|---|---|---|---|
| Passed maintainers' tests | 61% | About; it stopped the crash about 90% | 7 |
| Real security bugs tested | 112 bugs | 1 | |
| Open-source projects covered | 84 projects | C/C++, Go and Rust | 5 |
| Gain from build-and-test prompt | 20 points | On the C and C++ bugs | 16 |
| Benchmarks and datasets reviewed | 13 reviewed | 15 |
What happened
- The dataset covers bugs in 84 open-source projects written in C/C++, Go and Rust, and each AI patch was run against the tests the original developers added with the real fix.
- Across three Claude generations, the newest stopped the crash about 90% of the time but passed the maintainers' tests only about 61% of the time.
- One Claude Code agent with web search enabled retrieved the upstream fix commit in its first four tool calls, then reported a patch that took the same approach and cited the page.
- The team reviewed 13 benchmarks and datasets. Of the five built for AI agents, every one failed to address at least one of the nine pitfalls.
Why it matters
- exposure A team that merges on a crash-only pass would log the bug as fixed while the flaw stays in the code.
- decision A vendor's patching score now has four questions to answer: crash-only or maintainer tests, internet or git access, what the prompt said, and how timeouts were counted.
- constraint Build-and-test instructions alone moved C and C++ success by 20 points, so scores from tools prompted differently cannot be compared directly.
Those two rates imply that about 68% of the newest model's crash-stopping patches passed the maintainers' tests and about 32% failed [19]. That is close to the one-third reported for Opus 4.7 [2]. Across the 112 bugs [1], the gap between the two checks is 29 points [20], and Help Net Security's write-up says the best model failed the maintainers' tests on roughly four in ten bugs [8].
The crash check tests a single thing: whether the proof-of-concept input that triggers the bug still makes the program crash [3]. The PHP case in the write-up shows how a patch passes it. One patch fixes the logic the way PHP's maintainers did. The other throws an error whenever the inputs look unusual; that stops the crash but breaks calls PHP is supposed to handle [6]. Both pass PHP's existing tests, and only the test written for the fix separates them [6]. The authors call a fix of that kind "strictly worse than no fix at all," because it tells you the problem is solved when it is not [4].
The harness errors ran in both directions. Some inflated scores. One agent given the project's full git history searched for an earlier related fix and reproduced it almost line for line [10]. In an earlier version of the harness, another agent read a configuration file that pointed to the ground-truth patch, and its edits matched the upstream fix exactly [11]. The authors note that searching git history was sensible, and a human developer would have done the same; it was the benchmark that left the answer within reach [12].
Others deflated them. A hidden background check in Claude Code caused commands to stall. Those stalls passed for a model having trouble [13]. Elsewhere, an agent produced a correct fix along with a test of its own; that test clashed with the maintainers' test file, and the benchmark scored the correct fix as a failure [14]. The write-up does not say whether the 61% figure was rerun after those problems were fixed.
On a leaderboard that checks only whether the crash stops, every one of the patches that missed what the maintainers intended would have counted as a win [18].
What to watch
- Whether the authors rerun the 61% result with web search and git history closed off and the stall and test-clash errors fixed.
- Whether any of the five agent-focused benchmarks adds maintainer-test grading to close the pitfalls the team found.
- Whether remediation vendors publish answers to the four scoring questions alongside their patch-success numbers.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap+5
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Researchers at Northwestern University tested AI coding agents on 112 real security bugs.
- [2]
With Claude Opus 4.7, a third of the patches that made the crash go away still failed the maintainer tests.
- [3]
Many security patching benchmarks grade a fix on one question: does the proof-of-concept input, the sample that triggers the bug, still crash the program.
- [4]
A patch can pass the crash check by blocking the path to the crash and leaving the flaw where it was; the authors call that kind of fix "strictly worse than no fix at all," since it tells you the problem is solved when it isn't.
- [5]
The researchers built their dataset from bugs in 84 open-source projects written in C/C++, Go and Rust, and for each bug pulled the tests the original developers added alongside the real fix, then ran the AI patches against those tests.
- [6]
In a PHP bug, one patch fixes the logic as PHP's maintainers did, while another throws an error whenever the inputs look unusual, stopping the crash but breaking calls PHP is supposed to handle; both pass PHP's existing tests and only the test the maintainers wrote for the fix tells them apart.
- [7]
Across three generations of Claude models, the newest stopped the crash about 90% of the time but passed the maintainers' tests only about 61% of the time.
- [8]
Even the best model tested failed the maintainers' tests on roughly four in ten bugs.
- [9]
With web search left on, one Claude Code agent fetched the upstream fix commit within its first four tool calls, and later reported that its patch followed the same approach and cited the page it had fetched.
- [10]
Another agent, given the project's full git history, searched for an earlier related fix and reproduced it almost line for line.
- [11]
In an earlier version of the harness, an agent read a configuration file that pointed to the ground-truth patch, and its edits matched the upstream fix exactly.
- [12]
The authors point out that searching the git history was a reasonable move, the same thing a human developer would do, and that the benchmark had left the answer in reach.
- [13]
A hidden background check in Claude Code made commands stall, which looked like the model struggling.
- [14]
In one case an agent wrote a correct fix plus its own test, that test clashed with the maintainers' test file, and the benchmark scored the correct fix as a failure.
- [15]
The team reviewed 13 benchmarks and datasets used to evaluate AI patching; each of the five built for AI agents left at least one of the nine pitfalls unaddressed.
- [16]
Telling the AI how to build and test the code bumped its success rate on the C and C++ bugs by 20 points.
- [17]
Help Net Security lists four questions for evaluating an AI remediation tool: whether the score checked only that the crash stopped or also ran maintainers' tests; whether the agent had internet access or the project's git history; what the prompt said about building and testing; and how timeouts and pipeline errors were counted.
- [18]
On a leaderboard that only checks whether the crash stops, every patch that missed what the maintainers intended would have counted as a win.
- [19]
About 68% of the newest model's crash-stopping patches passed the maintainers' tests and about 32% failed.
- [20]
The newest model's pass rate falls 29 percentage points between the crash check and the maintainers' tests.
Sources
1 independent publisher whose own reporting we read for this story.
- helpnetsecurity.comAI vulnerability patching has a grading problem
1 article · October 11, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI vulnerability patchingFollow
- AI benchmark validityFollow
- AI Coding AgentsFollow