Lower-priced AI models ran up bigger bills in 32% of head-to-head tests

A rate card prices tokens. What a task costs depends on how many tokens the model decides to spend on it, and that count varies between models and between runs of the same model ^[20]. For Gemini 3 Flash's 80% discount to finish as a 38% premium over GPT-5.4, Flash had to bill about 6.9 times the token volume, assuming the discount held evenly across input and output ^[23]. The researchers traced much of that volume to thinking tokens, the hidden reasoning a model writes before it answers ^[18].

Agent work widens the gap. When a model works through tools or a computer environment, each turn can carry the earlier conversation back in as input, and in those tasks the number of turns drove cost ^[12]. On one prompt, Gemini 3.1 Pro finished in 85 steps for about $1. Gemini 3 Flash took nearly 1,000 steps, ran up $14 in token charges and failed ^[13]. That is roughly 12 times the steps for 14 times the money ^[24].

A finance lead could fairly read the headline result the other way, since the cheaper-listed model did not cost more in 230 of the 336 comparisons ^[22]. The 336 matches every pairing of the eight models on each of the 12 tasks ^[8][21], so each reversal belongs to one pair on one task. No model was consistently the cheapest or the most expensive; the ranking changed from task to task ^[9]. A buyer who knows the rate card was right about two times in three still cannot tell which of their own tasks sit in the other third.

Measuring on a team's own tasks helps more with choosing a model than with fixing a budget. The paper describes an irreducible noise floor: models can follow different reasoning paths even when their inputs stay fixed, and re-prompting does not remove the variation ^[2]. In a follow-up on two programming prompts, models from Anthropic, Google and OpenAI each got the same prompt five times and produced a different cost each time ^[14]. "The practical takeaway is clear," said Lingjiao Chen, a researcher at Stanford University and Microsoft Research ^[7]. "Price alone should not be used to infer which model is actually cheaper." ^[15] The team published its per-run cost data and code so companies can repeat the comparison on their own workloads ^[16].

Evidence that companies are missing their forecasts is of uneven quality. The survey comes from a vendor: Mavvrik sells AI cost-management software, and the figures are self-reported ^[19]. Uber's is a single company's account. Its chief technology officer, Praveen Neppalli Naga, used $1,200 of tokens in a two-hour demonstration of the company's coding tools in April ^[3], about $600 an hour ^[25]. "I'm back to the drawing board, because the budget I thought I would need is blown away already," Naga said ^[5]. The account of Uber's exhausted budget does not say how much of the overrun came from model choice and how much from volume ^[4].

The trade-off this quarter is between paying for a measurement run and picking from the rate card. The run costs tokens and engineering time now. Each task has to be repeated enough times to show the spread between cheap and expensive runs. I think the output worth paying for is a cost range per task type with its tail included, since that range is what next quarter's budget has to absorb. A team that skips the run sets that budget from a price comparison that reversed in 106 of 336 tests ^[6].