Coding agents solve 53-72% of real production tasks, and running the tests explains the spread

A team publishing on arxiv.org has released ProdCodeBench, a coding-agent benchmark built not from GitHub issues but from real sessions with a production AI coding assistant ^[1][2]. Across four foundation models, solve rates land between 53.2% and 72.2%, and the paper reports that the models making greater use of work validation tools - executing tests, invoking static analysis - are the ones scoring higher ^[5][7].

The construction is the interesting part. Each sample is a verbatim prompt from a real developer, the change that was actually committed, and a set of fail-to-pass tests: tests that fail before the change and pass after ^[3]. That gives an automated correctness signal without an LLM judge in the loop ^[10]. The set spans seven languages, chosen to reflect the language distribution of a large industrial codebase rather than a benchmark author's convenience ^[4]. Curation runs LLM-based task classification to find testable tasks, test relevance validation to confirm the tests actually exercise the changed code, and multi-run stability checks to drop flaky tests ^[9].

The motivation is a complaint about the alternatives, and it is a fair one. The paper notes that SWE-Bench draws tasks from Python GitHub issues, which diverge from industrial work in language distribution, prompt style, and codebase structure - standalone repositories versus monorepos, single repositories holding code for multiple projects on shared infrastructure ^[11][12]. The online options are worse for iteration speed: A/B testing takes weeks to reach significance, burns engineering time, and risks degrading the user experience while it runs, while shadow deployment avoids user harm but introduces non-determinism that undermines reproducibility ^[13][14]. Hence the stated design goals: verbatim prompts, the real task and language mix, and execution-based signals stable enough for both rapid experimentation and reinforcement learning ^[15].

On the numbers: the spread between best and worst is 19.0 points ^[16], and even the weakest of the four models leaves 46.8% of tasks unsolved ^[17]. Claude Opus 4.5 posted the highest solve rate in the paper's evaluation ^[6]. That ordering is less useful to an operator than the behavioural finding underneath it. The paper's reading is that iterative verification is what effective agent behaviour looks like, and that exposing codebase-specific verification mechanisms may significantly improve externally trained agents dropped into unfamiliar environments ^[8].

Worth keeping the causal arrow honest: as reported, this is an association between tool use and success, and the truncated material does not establish that forcing a weaker model to run tests would lift its score. But the practical implication holds either way. If tool use predicts outcomes, then an evaluation that only checks the final diff is discarding the signal that discriminates. Scoring whether an agent ran the suite, read the failure, and re-ran it is cheaper to measure than a new benchmark and closer to what you would demand of a junior engineer.

What to watch: whether the tool-usage analysis holds up per-model when the full paper is read, and whether other organisations take up the authors' invitation to build their own production-derived sets ^[18]. A benchmark harvested from one company's monorepo measures that monorepo. Several would start to measure agents.