Build1 publisher3 min readPublished
Routing by token purpose pays only once a gateway splits the call
A dev.to post credits a roughly 42x drop in token use on a code-editing workload to deleting an agent's reread loop. The gateway it recommends can only route whole calls, and the two levers are different sizes.
The Engineer · Build desk
What happened
- A dev.to post argues that deciding whether a whole request is hard is the wrong routing unit, because difficulty is a property you discover and not one you know before you send the request.
- It cites a recent write-up that cut token use roughly 42x on a code-editing workload by deleting the agent's explore-reread-redecide loop, with the model barely changed.
- In its place the post sorts tokens by job: cacheable instruction and system tokens, routine extraction and formatting, a cheap first pass with escalation on failure, and a genuinely ambiguous slice held on frontier.
- It puts routine work on models it says are typically 70%+ cheaper, and reports that splitting a classify-draft-validate support pipeline this way typically drops the bill 70%+, or 90%+ on China models.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Adopting escalate-after-failure means funding a validator that can tell a bad cheap pass from a good one, because a gate that never fires leaves the hard slice on the cheap route.
- contradiction The post's headline anecdote is a token-count saving while its savings figures are unit-price savings, and quoting them together implies a combined effect neither one supports.
- exposure Sending every job through one gateway puts the cost story inside that vendor's telemetry, so the buyer's only check on the blend is whatever request-level tracing the vendor exposes.
Two levers sit next to each other in that post and they are not the same size. Deleting a deliberation loop reduces the number of tokens. Picking a cheaper model reduces the price of each one. The post puts a ceiling on the second: routine extraction and classification run on models it describes as typically 70%+ cheaper, and up to 90%+ cheaper on China models [5]. Ninety percent off is a 10x cut in unit price. A 42x cut in token use is 97.6% fewer tokens [15]. At the deepest discount quoted, you still have to find 4.2x fewer tokens to get to 42x [16].
Where the split happens matters more than the ratio. The post is direct about it, saying you cannot easily separate a prompt's formatting tokens from its reasoning tokens before you send it, and that the call boundary is where that separation happens [8]. A gateway at that boundary sees a request and assigns it a route [9]. It sees calls. If your agent does classification, drafting and schema validation inside one call, that call gets one route, and the mechanical 80% the post describes travels with the hard 20% [20]. Making the sub-jobs routable means emitting them as separate calls with their own prompts, schemas and retry handling. That work happens in your code, upstream of any gateway.
The escalation gate has arithmetic of its own. Run a cheap first pass and escalate only the part that failed [4], and you pay twice on whatever escalates. Put the cheap route at 10% of frontier price [5] and call the escalation rate f. Blended cost against a frontier-only baseline of 1 is 0.1 plus f. To reach the 70%+ drop the post reports for a classify-draft-validate support pipeline [7], f has to stay at or below 0.2 [17]. The gate also needs a check that actually fires: schema validation catches malformed output, and a well-formed wrong answer passes.
The claim that cheaper models are indistinguishable in output on routine work [5] is a claim about someone else's tickets. For it to transfer, your label set has to be as separable as theirs, your schema as tight, and your parse-failure rate similar. Each of those pushes f the wrong way when it differs. The post's estimate that extraction and formatting alone might run six figures a year at frontier prices is a range with no floor [6].
The 42x number is second-hand. The post credits it to a recent write-up it does not name or link [18], so the workload, the harness and the baseline are not available to check. The post also closes on a product: an SG-hosted gateway on Tencent Cloud, PDPA-aligned, one OpenAI-compatible endpoint over 25+ models, aimed at teams in Malaysia, Singapore and Indonesia working under data-sovereignty rules [13]. The instrumentation it asks for is the part worth keeping: cost per successful task including retries and escalations, with request-level tracing [11]. "A gateway that hides the blend is just a black box you're told is cheaper," the post says [10].
What to watch
- Publication of the write-up behind the 42x figure, with its harness and baseline, which would make the loop-deletion claim checkable.
- Escalation rates from a real deployment, reported as cost per successful task including retries, which is the only test of the 70%+ claim.
- Whether gateways start reporting which route each sub-call took, and not only a request-level total.