Loading today’s stories
Build1 distinct publisher3 min readUpdated
SWE-Bench ProMax puts frontier coding agents on 170 curated refactoring commits. The number that should move procurement is a different one: nearly 60% of unsolved SWE-bench Verified tasks have flawed tests.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Researchers at Shanghai Jiao Tong University, Peking University, Douyin Group and other institutions have published SWE-Bench ProMax, a multilingual refactoring benchmark of 170 instances taken from real commits across seven languages, and the best model managed a 41.2% resolve rate on it [1][2][3]. If you are buying agents on the strength of a SWE-bench score, the more consequential line in the writeup is the audit the benchmark's authors cite: nearly 60% of unsolved SWE-bench Verified instances contain flawed tests [4].
Convert the headline figure into work and it gets concrete. At 41.2% of 170 instances, the best model clears roughly 70 tasks and leaves about 100 on the table [5]. The instances are not scraped and shipped: the creators say every one went through multi-stage curation, with rewritten issue descriptions, manual review of test suites to strip out tests that were too narrow or too broad, and removal of tasks that lacked complexity or cross-file scope [6]. That last filter is the design choice that matters, because it is what makes the set a refactoring benchmark rather than a bug-fix benchmark. The researchers describe the result as a meaningful and unsaturated challenge for current agents [7].
Why refactoring in particular. Shane Warden, principal architect at ActiveState, told The New Stack that strict refactoring "demands zero tolerance for error, zero tolerance for behavior changes, and complete reversibility" [8]. His objection to the prevailing method is structural, not stylistic: engineers who treat models as text-processing engines feed a whole codebase into a large context window, prompt, and wait for multi-file diffs [9], an approach that assumes deep understanding of large systems. "I don't believe that premise," Warden said. "I believe that token proximity does not guarantee structural understanding" [10].
Vojtech Pavlik, senior director of technical strategy for core infrastructure at SUSE, put the ceiling more bluntly, saying there are no LLMs that can read a large codebase and understand it all at once, and that the capability is "still far out of reach" [11]. He added a mechanical reason: very large models need highly optimized attention algorithms such as DeepSeek Sparse Attention to be usable at all, and on a large or convoluted codebase that maxes out the context window, that can mean missing important observations [12]. His second failure mode is the one that does not show up in a resolve rate at all. Where code meets time, he said, you get race conditions, lost idempotency, lost atomicity and incorrect retry handling: code that passes all tests and works if the user follows the spec exactly [13].
The procurement problem sits on top of all this. Evaluation quality for coding agents is considered to be in decline, partly because frontier models can sail through benchmarks whose solutions have leaked into training sets [14].
What to watch: whether SWE-Bench ProMax stays unsaturated once vendors start targeting it, and whether anyone publishes per-language results. The source does not give the language split, and 170 instances over seven languages averages about 24 each, which is thin ground for claiming an agent is good at Rust or C++ [15].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
AI coding agents still struggle with large-scale refactoring, with the best model achieving only a 41.2% resolve rate on a new refactoring-focused benchmark developed by researchers at Shanghai Jiao Tong University, Peking University, Douyin Group, and other institutions.
A recent audit cited by the researchers shows nearly 60% of unsolved SWE-bench Verified instances contain flawed tests.
Per its creators, every instance underwent multi-stage curation: they wrote issue descriptions to make them more precise, manually reviewed test suites to remove overly narrow or broad tests, and filtered out tasks with insufficient complexity or limited cross-file scope.
For AI coding agents, evaluation quality is considered to be on the decline, as frontier models can sail through benchmarks if solutions have leaked into training sets.
SWE-Bench ProMax is a multilingual code refactoring benchmark of 170 instances drawn from real commits.
The benchmark spans seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single-source trade reporting on a creator-reported benchmark
Everything rests on one article from one publisher. The headline 41.2% figure and the curation methodology are creator self-reports relayed without a link to the paper, leaderboard, model identities or run dates, and the nearly-60% flawed-test statistic arrives third-hand via the researchers' citation of an unnamed audit. The expert material is well attributed and specific but is practitioner opinion, not measurement. Only the arithmetic derivation is fully checkable.
Benchmark published and run once; no third-party uptake shown
The observable adoption events are the benchmark's existence and an evaluation of frontier models on it. There is no evidence of any lab, vendor or engineering organization using SWE-Bench ProMax in its own reporting, no leaderboard, no repository or license information, and no disclosure of who ran the models. Nothing in the source supports uptake beyond the authoring group.
Sober framing, but precise numbers outrun their verification
The article is deflationary in tone — its thesis is that agents fail at refactoring and that benchmarks are unreliable — which limits overstatement. The mild positive gap comes from presenting two decimal-precise, unverified figures (41.2% and nearly 60%) plus a self-reported claim of superior curation as settled findings, and from asserting a general decline in evaluation quality without any measurement. The counterweight is that the strongest practical warning, tests passing while concurrency behavior breaks, is understated relative to its importance.
Benchmark authors and vendor architects both have stakes
The benchmark's creators, including a commercial participant in Douyin Group, benefit from ProMax being seen as the harder, cleaner successor to SWE-bench, and the article carries their curation and unsaturated-challenge claims unaudited. The two named experts are employed by commercial vendors (ActiveState, SUSE) whose positioning favors skepticism toward whole-codebase LLM refactoring; the article does not disclose or examine those interests. The publisher is a developer-tooling trade outlet whose audience rewards this framing.
Directionally credible, numerically fragile
The qualitative core — that large-scale cross-file refactoring remains unsolved for current agents and that benchmark test quality is a real problem — is coherent, specifically attributed and consistent across both experts. Confidence is held down by the single-publisher cluster, the absence of the primary paper and audit, undisclosed model identities behind the headline score, and zero evidence of adoption beyond the authoring group.
build
Codex can now ask and keep going, which deletes the only checkpoint you were getting for free1 distinct publisher
build
2,513 tool calls, zero refactorings: what agents actually do when you ask them to refactor1 distinct publisher
build
A green build only proves your agent was consistent with itself1 distinct publisher
leadership
The AI bill nobody reconciles: cost per finished task, not per million tokens1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
1 article · August 21, 2026