Arize's cheapest model per finished task reliably solves only a fifth of the benchmark

The retry tax is the part of the argument most buyers expect to matter, and in this study it did not. The pass rate on gpt-oss-120b implies about three attempts for every completed task ^[2], and those attempts are already inside its price. Arize counted everything a model spent on every attempt, including runs that failed, retried or timed out, divided by the number of completions ^[3]. Token price cannot see retries, failed tool calls, malformed outputs or runs that grind to a limit and never finish, the post says ^[19]. Applying the study's 23x comparison to gemini-3.5-flash puts that model near $1.24 for the same finished job ^[1].

What caps the cheap model is coverage. Twenty-five reliable tasks minus eight is seventeen, so a wholesale substitution moves a product from 62.5% of the suite to 20% ^[3]. Arize wrote that if a team makes that swap, "your bill collapses and so does the set of things your product can do" ^[14].

The saving is real on easy work only. On easy tasks gpt-oss-120b passed 65% against GPT-5.5's 69% and Kimi K2.6's 73%, at roughly a twenty-fourth of the cost per attempt ^[7]. On hard tasks only the top tier competed, and the cost-effective models fell far behind; Arize's reading is that the hardest problems take a frontier-class model whether it is open or closed ^[8]. On retrying as a substitute for capability, the post is blunt: "a model that cannot do a task does not learn it on the fourth attempt, it just bills you four times" ^[9].

The two frontier-class models split their strengths by difficulty. Kimi K3 solved every easy task in the study, all six trials of all of them, where GPT-5.5 slipped to 69% on that tier ^[11]. On the hardest tasks the order reversed, GPT-5.5 at 51% and K3 at 32% ^[12].

The obvious objection is provenance: this is a benchmark run and published by Arize and Fireworks ^[1]. The result that carries the procurement argument works against the cheap-model headline and sits in the same post, which is a reason to take the coverage count seriously ^[13]. Two limits belong with it. Each model got six trials per task ^[2], a thin base for separating a 33% pass rate from a 40% one. Arize also said it did not test GPT-5.6 sol and that these results should not be read as a comparison with it ^[16].

The case for changing the metric rests on a pricing claim. According to the post, the first few years of the LLM era ran on heavily subsidized pricing set to capture market share instead of covering cost, and that era is ending ^[17]. If that holds, the distance between a vendor's price page and the cost of a completed job widens. The difficulty mix of a team's own queue tells a buyer more about that cost than the label on the model does. A team that reprices agent work on finished jobs this quarter owns a routing decision next quarter, and Arize's answer to which model fits is to "measure it on your workload" ^[18].