Build1 distinct publisher3 min readPublished
JetBrains put two frontier models through the same coding agent, which scored them as a tie on tasks solved even though the runs differed by 47% in steps and 2.25x in dollars, and that gap is what procurement actually pays.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Dividing the dollars by the steps shows where the tie actually comes apart. Opus works out to roughly USD 0.0152 per step and Gemini to roughly USD 0.0046, a ratio of about 3.3x on one loop iteration [4]. That is not the longer profile doing less per dollar: it is buying 87 more attempts at under a third of the unit price [1][4].
What a step buys depends on the harness. Junie takes an issue and a repository, then lets the model inspect files, search for symbols, edit code, run commands, and execute tests [6]. So a step is one inference round trip plus one tool call against your codebase. If that call is a symbol search, 87 extra of them cost seconds. If it runs a full test suite on your CI, the token bill is the smaller line item.
Transfer needs the same tool surface, since steps are counted against Junie's action set rather than the model's. It needs a similar context policy, because per-step cost tracks how much history gets resent each turn. And it needs a similar task mix: the JetBrains Research post cited in the writeup warns that gains may not transfer to other tasks even inside the same codebase, and that rankings move with task type [10]. The writeup reports steps and dollars for this pair while defining efficiency to also cover tokens, model calls, tool calls and seconds [8], so a reader outside JetBrains cannot turn the profile into a latency estimate.
The part worth copying is the failure taxonomy. A run can fail because the agent never found the relevant code, misread the cause, edited the wrong layer, implemented only part of the fix, or stopped without adequate validation [15]. Those are different bugs with different owners. A pattern of repeated searching points at repository navigation or tighter prompting [12]; a correct diagnosis attached to an incomplete patch points at implementation or task completion [13]. Resolve rate collapses all of that into a percentage of tests passed [5], and a percentage has no opinion about who pays for the run.
Patch scope is the other cost that lands on a person. Two patches can implement the required behavior and still differ in scope, complexity and fit: one edits a single relevant function, the other adds helpers, state, branches and unrelated files, and both go green [11]. The second one costs a reviewer an afternoon. Trajectory data is what lets you see it coming, because the trace shows whether the model localized the problem before editing, repeated the same searches, tested its assumptions, and kept the patch contained [7]. None of that requires access to the model's private reasoning [7]. It requires logging the tool calls you already make.
Ranked by verification strength, evidence, and original report placement.
On one of JetBrains' private benchmarks, Claude Opus 4.7 and Gemini 3.5 Flash solved the same number of tasks.
The pipeline's efficiency perspective asks how many tokens, model calls, tool calls, and seconds a run required, and what it cost.
For each benchmark task the pipeline combines the issue, repository context, generated patch, test result and execution trace, and evaluates the run from four perspectives: outcome, efficiency, patch quality, and process quality.
A JetBrains Research post describes a "benchmark meaning gap" identified in a recent research paper: a benchmark measures performance under a particular setup, but its score is often treated as evidence of much broader coding ability; performance gains may not transfer to other tasks even within the same codebase, and model rankings can change with the task type.
The post "From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding" was published on JetBrains' Junie blog.
Opus used an average of 184 steps and cost USD 2.79 per run.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · August 31, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
2,513 tool calls, zero refactorings: what agents actually do when you ask them to refactor1 distinct publisher
leadership
Cost per successful task, not per token: a 2,400-run benchmark reorders the model shortlist1 distinct publisher
build
Junie Local is free. The 64 GB M5 Mac is the price.2 distinct publishers
build
Claude Code now outruns Copilot roughly two to one in JetBrains' survey of 15,000 developers1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Specific numbers, one closed source
The figures are precise and internally consistent — 184 versus 271 steps, USD 2.79 versus USD 1.24, 523 tasks across four datasets — and the method behind them is described in more detail than vendor engineering posts usually bother with. But the benchmark is private, the datasets are unnamed, no resolve rate or variance accompanies the tie, and the write-up we have breaks off just as the trajectory comparison begins. Nobody outside JetBrains can rerun any of it.
One team's internal instrument
Trajectory-level scoring exists here as in-house tooling at exactly one company, applied to one model bake-off. There is no open release, no customer using these profiles to choose a model, no other lab reporting step-and-cost profiles alongside resolve rates. The practice the post argues for has, so far, precisely one practitioner.
Modestly ahead of what it shows
The prose itself is careful — it concedes that long trajectories can be legitimate, that repeated file reads may be necessary, that benchmark scores travel badly. The stretch is structural rather than rhetorical: a title promising model profiles and a pipeline pitched as the answer to leaderboards, resting on one closed benchmark whose results the reader never sees in full. Two step averages and two prices do not yet constitute a profile.
The agent's vendor grading the graders
JetBrains ships Junie, and this post argues that the metric everyone uses to rank coding agents is inadequate while the richer view happens to require the kind of trajectory data Junie produces. That is a real commercial interest and it goes undeclared. Working against it: the comparison flatters neither of JetBrains' model suppliers, the cheaper model comes out looking good, and the failure taxonomy is written to be useful rather than flattering.
Directionally trustworthy, thinly held
We are confident about the arithmetic and about what JetBrains said; we are not confident the numbers would survive an outside rerun, and the text stops before the comparison it promises. Everything traces to one publisher with a stake in the conclusion, so treat the 2.25x cost spread as a live hypothesis about your own workload rather than a settled measurement.