Build1 publisher2 min readPublished
A rerun of the same SWE-bench task can consume 30 times the tokens
A study of eight frontier models on SWE-bench Verified puts agentic coding at roughly 1,000 times the token cost of code chat, dominated by input, with the models' own pre-run estimates correlating no better than 0.39.
The Engineer · Build desk
What happened
- Researchers ran eight frontier models through the OpenHands agent on SWE-bench Verified, measured where the tokens went on each trajectory, and then asked the models to forecast their own costs.
- Agentic coding tasks consumed 1,000 times more tokens than the code chat and code reasoning workloads the paper compares them against.
- Input tokens, not output tokens, drove the overall cost, and the paper reports that holding even with token caching enabled.
- Runs of the same task differed by up to 30 times in total tokens, which the authors describe as inherently stochastic variation in usage.
- Asked to estimate their token cost before execution, the models correlated only weakly to moderately with the outcome and systematically guessed low.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A per-task budget calibrated on one observed run has no dependable relationship to the next run of the same task, so the cap has to be enforced during execution or set high enough to cover the tail.
- exposure Any budgeting feature that asks the model to estimate its own spend inherits a documented downward bias, and the customer absorbs the overshoot.
- decision If accuracy peaks at intermediate cost and flattens above it, a token ceiling is not automatically an accuracy penalty, and the ceiling becomes a parameter worth varying.
- cost Under the pricing the paper describes, a failed task still bills, so both the variance and the failures land on the buyer's invoice.
A tool-calling loop bills for what it reads. Each file read and each test log joins the conversation history, and that history goes back to the model on the next turn. Instructions to be terse, and output token caps, act on the smaller half of the bill [3].
Take the 30x figure literally and the expensive run of a task costs 2,900 percent more than the cheap one [13]. A per-task budget has to absorb that spread. The paper gives 30x as an upper bound on the runs it saw, so there is no percentile to read off it [4].
The models' own estimates leave that gap where it is. The strongest correlation the paper reports between a model's pre-execution estimate and its actual usage is 0.39 [6]. Squared, that is 0.15, so in the best case reported the estimate accounts for about 15 percent of the variation in real spend [14]. The estimates also skew low [6]. Like most software estimation, they guess under the real bill. That leaves human judgement, and the paper finds expert difficulty ratings align only weakly with actual token cost [7].
The authors call this the first systematic study of token consumption in agentic coding [12], and the setup is one scaffold on one benchmark, with eight frontier models driving OpenHands on SWE-bench Verified [1]. The 1,000x multiplier compares those trajectories against chat and reasoning figures drawn from separate cited sources [8]. For it to transfer to your stack, your agent would need a similar number of turns per task and similar repository reads per turn. Your billing would also have to treat cached input the way theirs did.
Model choice is the one input you have before a run starts: on the same tasks, Kimi-K2 and Claude-Sonnet-4.5 average over 1.5 million more tokens than GPT-5 [9]. The authors' stated motivation for up-front estimation is that users could choose models accordingly and providers could design clearer pricing tiers, enforce budget caps and trigger early alerts for large bills [10]. Caps can be enforced without a forecast. The paper does not supply one, so the enforceable point is mid-run, against tokens already spent.
What to watch
- Whether an external predictor, trained on trajectories, beats the models' own 0.39 correlation on the same tasks.
- Whether any coding agent vendor ships a mid-run token hard stop instead of a post-run invoice.
- A replication on a different scaffold, to test whether input dominance and the 30x spread survive outside OpenHands on SWE-bench Verified.