Skip to content

Invest1 publisher3 min readPublished

A $14.34 router matched Opus-5's score on LiteLLM's 21-task benchmark

LiteLLM's own Terminal-Bench run puts a gpt-5.4-mini classifier routing across Haiku, Sonnet and Opus at $14.34 against Opus-5's $19.74 for the same 16 solved tasks. The saving works out at 26 cents a task.

The Investor · Invest desk

Illustration accompanying A $14.34 router matched Opus-5's score on LiteLLM's 21-task benchmark

What happened

  • A router with a gpt-5.4-mini classifier reading only the current user message solved 16 of 21 Terminal-Bench 2.0 tasks for $14.34, the same count Claude Opus-5 solved for $19.74.
  • Giving a 4o-mini classifier the last three user messages instead of one lifted its solve rate from 66.7% to 76.2% and its bill from $12.86 to $18.58, a 44% cost increase for 14% relative quality.
  • The heuristic router, which makes no classifier call at all, reached 87% of Opus quality at 56% of the cost and had the lowest median wall clock of any of the eight arms.

Compiled by The InvestorSomething wrong?How this is made

Why it matters

  • constraint At 26 cents saved a task, a buyer needs roughly 3.9 million tasks a year for routing to be worth $1m, so this is a lever for gateways and large agent fleets and close to noise for a team running a few thousand jobs.
  • decision Anyone budgeting classifier spend now has a priced comparison inside one run: paying for a better classifier model got the same 16 solved tasks for $4.24 less than paying for a wider classifier context window.
  • exposure The only figures on the table belong to the vendor selling the router, published on its own documentation site with a design-partner solicitation attached, so procurement teams modelling savings are modelling one party's spend logs.

The saving is $5.40 [1]. Spread across 21 tasks that is about 26 cents a task [2], and the classifier call that produced it cost at most $0.34, which is 2.4% of the winning arm's bill [8][11]. Per solved task, the router paid $0.90 and Opus-5 on its own paid $1.23 [4]. Turning 26 cents a task into $1m of annual saving takes about 3.9 million tasks [3].

The context result is narrower than it sounds. All three context arms use one 4o-mini classifier and one tier map, so the only variable is how much of the conversation the classifier reads before it picks a tier [15]; the answering model's context window is not what moved. Reading the last three user messages instead of one raised Opus's share of requests from 34% to 45%, and LiteLLM attributes most of the extra cost to that escalation [9]. The bill went from $12.86 to $18.58 [3]. Swapping in gpt-5.4-mini on a single message bought the same 16 tasks for $14.34, or $4.24 less [6]. Median time per task rose from 6.8 minutes to 8.3 [4]. The arm that also read assistant replies sent 5% of requests to Haiku and scored worst of the three [6].

The arm I would look at hardest has no classifier in it. Its 87% of Opus quality at 56% of the cost [7] works out to roughly $11.05 and about 14 of 21 tasks [7][8], which is $0.79 per solved task, cheaper than the routed or the unrouted Opus figure [9]. The two extra tasks the gpt-5.4-mini router picks up cost $3.29 between them, $1.65 each [10]. LiteLLM reports cost as a per-arm total across all 21 tasks, taken from its gateway spend logs [14].

The scoring is soft at this size. One task out of 21 moves a solve rate by 4.8 points [12], and errors ran between zero and four per arm, excluded from the denominator only where a task did not complete for that arm [11]. Every tier is Anthropic: Haiku for SIMPLE, Sonnet for MEDIUM, Opus for COMPLEX and REASONING, with Sonnet as the fallback [10]. All eight arms ran the same 21 tasks with prompt caching on, which is what LiteLLM means by like for like [12][1], and the company published the run on its own documentation site alongside a call for design partners [13].

The router cannot beat Opus, because its top tier is Opus, reached on between 34% and 45% of requests [9]. So what is on offer is a price effect on a fixed ceiling. The claim most likely to break is the one about the single message: Terminal-Bench tasks arrive as self-contained specifications, so one user turn tells a good classifier what it is dealing with, and on a workload where the first message does not describe the job, the arm that paid $18.58 for three turns of context is the one that should win [3][15].

What to watch

  • Third-party numbers posted to LiteLLM's discussion #32168 by teams running the router on their own production traffic.
  • Any price change across claude-haiku-4-5, claude-sonnet-5 and claude-opus-5, since the whole saving is the spread between those three tiers.
  • Whether LiteLLM publishes task-level results per arm, so the failures of the heuristic and classifier routers can be compared directly.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories