Trail of Bits says 1Password's 26% AI patch score reflects flawed prompts, no-code-execution trials, and grading errors, not true AI performance

The 26% is an average across conditions a defender would not set up. Two prompt variants told the agent to apply the wrong fix, and those runs are 22% of the data, according to Trail of Bits ^[3]. A separate evaluation mode stopped agents from building or running code, and that mode is another 36% ^[4]. Combining untested patches with tested ones makes the headline partly a count of how often the agent could not build or run the code.

The six vulnerabilities were selected because their fixes were complex ^[5]. Clean-fix rates across those bugs ran from 3% to 60%, so the average depends heavily on which bugs made the list ^[6].

The grading is where the report argues against itself. The two models assigned different outcomes to 36.8% of the same patches, and the headline averages their assessments ^[9]. Models grading their own patches agreed with human reviewers on the full five-category outcome in 65.9% of reviewed cases ^[10], which leaves human and machine verdicts apart on 34.1% of them ^[3]. Errors run in both directions. The authors found 248 generated patches that repeated an off-by-one error present in the upstream Linux reference fix, and the automated grader caught that new vulnerability in 24 of them ^[11] - it missed 224, under 10% caught ^[1][2]. On ActiveMQ, 8% of verdicts scored an intended behavior change as a regression ^[8]. The Chromium grader marked many patches clean while they still left a use-after-free in a callback ^[12].

Trail of Bits reanalyzed the published patches and test results, keeping trials where agents could run code and were not instructed to apply the wrong fix, and excluding runs the study classified as having consulted the upstream fix. In that subset, 2,634 of 3,067 patches blocked the supplied exploit ^[7]. Trail of Bits says blocking the exploit does not establish a complete repair ^[13].

The critique comes from an interested party. Trail of Bits published it alongside two agent skills, post-patch-validation and review-walkthrough ^[14], and says it wants its work to help defenders fix more vulnerabilities ^[15]. The post does not include a response from 1Password. The reanalysis was possible because 1Password shipped its code, patches and test results with the study.

One more design choice sits under the number: GPT-5.5 ran at medium reasoning effort and Opus 4.8 at high, both tool defaults, and the authors did not measure what raising effort did ^[16].

That defenders are already acting on the 26% and leaving repairable bugs alone is Trail of Bits' inference ^[2], and the design problems it describes are in the data. A team that has barred agents from patch work should know the number it cited measures instructed sabotage and untested edits alongside ordinary repair attempts.