JetBrains' Kotlin leaderboard prices the same 86 solved tasks at 3.7x apart

Start with the two entries that tied. Codex with GPT 5.5 xHigh solved 86 tasks on 38.78M tokens, or about 451,000 per solved task ^[9]. Junie with Opus 4.7 max solved the same 86 on 19.98M tokens, about 232,000 each, and took 19 hours 58 minutes of wall clock ^[8]. The token bill differs by 3.7x ^[10].

The gap comes from the loop, not the weights. Each task hands the agent a real issue description, and the agent has to navigate the project and produce a patch that passes the repository's own tests inside a containerised environment ^[3][4]. Every wrong file read, every re-listed directory, every retry after a failing test run is billed input. None of that shows up in the resolution rate; it shows up in tokens per solved task. A harness that re-reads context on each turn pays for the same source file many times.

The reasoning-effort dial shows the same thing in one product family. Claude Code with Opus 4.7 medium solved 80 tasks on 5.31M tokens, about 66,000 per task ^[6]. The xhigh setting solved 90 on 10.59M tokens, about 118,000 per task ^[7]. The delta is 10 extra tasks for 5.28M extra tokens ^[11], which is 528,000 tokens per marginal task ^[12], roughly eight times the medium setup's average ^[13]. That premium is worth paying when the last ten tasks are the ones blocking a release. It is not worth paying on a backlog of small fixes, because medium already solved those at a sixth of the marginal rate.

The cheap-model assumption fails here too. Junie with Gemini 3 Flash solved 64 tasks and burned 40.17M tokens, about 628,000 per solved task ^[14]. Gemini CLI with Gemini 3 Flash finished last on quality at 47 tasks and 777,000 tokens per solved task ^[15], about 12x the per-task cost of the Opus 4.7 medium entry ^[16].

For these numbers to transfer to your repository, your tasks would need the same difficulty distribution as 105 issues drawn from active open-source Kotlin projects ^[3][2]. Your prompts and tool configuration would need to match what JetBrains ran, since the harness is what the 3.7x measures. And your economics would need tokens to be the binding constraint. Wall clock is a separate column: the same Codex 5.5 run finished in 8 hours 30 minutes while the more token-frugal Junie entry took nearly 20 ^[9][8]. If engineer waiting time is what you are paying for, the ranking flips.

The benchmark itself is honest work. It follows SWE-bench methodology at repository level ^[3], runs on the open-source Multi-SWE-bench infrastructure, and JetBrains published the datasets, test harnesses and leaderboard along with a methodology page and the GitHub repository ^[5]. The dev.to writeup that resorted the board by tokens per solved task states plainly that the author has not run the evaluations and that the token math is theirs, drawn from the official leaderboard and announcement post ^[17]. The leaderboard reports tokens, not dollars, so converting a Flash loop's 40.17M tokens into a comparable spend against an Opus loop is still work you have to do yourself ^[14].