Smart Routing cost 35.6 times as much as a cheap fixed model in LLM Gateway's own benchmark

On the write-up's cost index, the premium arm's total is 100 ^[15]. Smart Routing comes to roughly 66.8 ^[1]. The mid-priced model comes to about 18.1, and the low-cost model to about 1.9 ^[3][2]. The router's total includes the separate Jev classifier charge and its storage ^[15]. That fee comes to about 0.35 index points, roughly a fifth of what the low-cost arm spent on everything ^[4]. Almost all of the router's cost is the models it chose ^[16].

The router picked from a pool that held both the low-cost and the premium model ^[9]. It can also raise reasoning effort on requests it rates as hard. The first run capped output at 4,096 tokens, and the team found that cap blocked the escalation, so they finished the round and repeated all of it at 8,192 tokens ^[13]. The primary figures come from the rerun, with the router free to spend tokens. The summary does not give per-arm pass rates or the split between cheap and premium picks. The published results record the selected model and score for every call ^[3].

On quality, the write-up says the cheaper fixed baselines "remained competitive on task success" ^[10]. The task set is a subset of Google Research's IFEval. It checks instructions such as avoiding commas or formatting an answer as JSON, and it does not establish factual accuracy or writing quality ^[11]. The gap to the low-cost arm carries over to another team's traffic only if two things are true. The traffic has to be graded by rules like these. The team's cheap model also has to pass them about as often as this one did. Where the failures are wrong facts or weak prose, the router's escalations may be buying something this test does not score.

The method is careful. Prompts were fixed before any results were collected, and prompt and arm order were shuffled with a recorded seed ^[2][3]. Failures and truncated responses stayed in the denominator ^[19]. Classifier charges came from the actual request records ^[15]. The authors audited their own scorer. They flagged one IFEval prompt that contradicts itself and five case-conversion prompts with ambiguous trailing punctuation ^[19]. They checked stored outputs against the official evaluator and verified all 15 exact-answer reference solutions ^[12]. The author works on LLM Gateway, says so, and notes the republication was prepared with AI assistance ^[1]. The write-up still says of the premium comparison: "It includes no savings, so this small sample does not establish a reliable reduction." ^[7]

Requests ran one at a time, with gateway response caching and cross-provider fallback disabled ^[14]. The fixed arms did not pin a provider. Provider-side prompt caching could still have occurred, and reasoning and temperature stayed at their defaults ^[14]. Each arm saw 80 requests per output cap ^[6]. Latency was measured at the client and includes network and gateway delay, and some client timings ran well past the logged inference duration ^[17].

In my view, the fixed cheap model is the first arm to run on instruction-following traffic. After that, routing has to justify its cost on the prompts the cheap model fails, and the per-call scores in the published results make those prompts countable ^[3].