Build1 distinct publisher3 min readPublished
One team's forty lines of Go moved 81% of requests to a cheaper model and cut the bill 71%, but the arithmetic says the win came from a boring traffic mix and a wide price gap, and the bill it pays is tail latency.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The two published figures reconcile in only one way, and which way changes what you copy. Price a hundred requests under the old setup: all strong, at about $0.0145 a call, $1.45 [9]. Now read the 81% as the share that *attempts* the cheap model first, with 15% of those escalating [3][11]. That is 81 cheap calls, 12.15 strong retries, and 19 requests pinned to the strong tier: $0.565 per hundred, a 61% cut. Read it as the share *finally served* by the cheap answer and it lands: 95.3 cheap attempts, 14.3 escalations, 4.7 requests sent straight to strong, $0.409 per hundred, a 71.8% reduction [20]. Under that second reading, the `ForceStrong` and `alwaysStrong` short-circuits at the top of the router cover under 5% of production traffic [17][20]. The post does not say which reading it means.
The cheap side does check out against the posted prices. Three thousand input tokens at $0.20 per million is $0.0006; 700 output at $1.20 is $0.00084; total $0.00144, which is the post's $0.0014 [19]. The strong model is never named in the excerpt, so its $0.0145 per call is a number you cannot audit. Flattened across the same 3,700 tokens it implies roughly $3.92 per million [21]. If your two tiers sit closer than that, the break-even algebra is the first thing to redo.
Escalation costs $0.0159 against $0.0145, about 10% more than going strong directly [10], which is why the money question closes fast and the gate becomes the whole design. The `Gate` type returns the reason an answer cannot be trusted, or empty string to accept it [13]. The first default gate is `SchemaInvalid`: the response had to be JSON matching a schema, and wasn't [14]. That check is free CPU. A gate that needs a model to score the output re-imports the exact per-request tax that killed the first design: another call, another 300ms [5].
The classifier failed on something worse than its cost, though. Length correlated with the shape of a request and not its difficulty: "Fix the timezone bug" needs everything you have, while "Summarise the following 4,000-word RFC" is long and trivial [7]. The post calls the first of those eleven characters; it is twenty [22]. The point survives the miscount. Difficulty is a property of the work, so the only honest way to learn it is to do the work [8].
The error handling is the part I would keep verbatim. When the cheap call errors, the router falls through to strong, because a 429 or a timeout is availability and not quality. When the strong call errors after escalation, it returns the cheap answer: degraded rather than failed [15]. That second line is a policy decision hiding in a return statement. It means a strong-tier outage reaches users as slightly worse answers instead of errors, and the only place it shows up is the escalate and accept counters, which is why those carry a reason label [18]. The escalation call also resends the original request, not the original plus the cheap draft; the author flags that as failure mode four and the one people get wrong, and the excerpt stops before the other three arrive [16][4].
This transfers if your traffic really is mostly titles, diff summaries and extracted fields [2], and if your gates can be assertions rather than judgments. Both are checkable before you write the router.
Ranked by verification strength, evidence, and original report placement.
The Gate type is documented as returning the reason this answer cannot be trusted, or empty string to accept it.
The first entry in defaultGates is SchemaInvalid, commented as: had to be JSON matching a schema, and wasn't.
The overwhelming majority of the team's requests were tasks like titling a thread, summarising a diff, extracting fields from a form, and naming a file, so frontier prices were being paid to generate three-word document titles.
The author states the change broke four things, and calls those failures the interesting part; the supplied excerpt ends inside the gate list before all four are enumerated.
The first design was a classifier in front of the router that decided whether a request was easy or hard; it was built and rejected partly because every request then paid an extra call before any work happened, a cheap classifier adding another Luna call and another 300ms while an accurate classifier meant a strong-model call.
The working design is: send the request to the cheap model, look at what came back, and if it fails a gate, run it again on the strong model and return that instead.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 29, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
invest
Nvidia's Perplexity talks move its money one layer further from its own chips1 distinct publisher
build
OpenAI's top model at $4/$20 is a three-month answer to a permanent build decision1 distinct publisher
build
Bedrock turns GPT-5.6 throughput into a routing choice, with residency as the price1 distinct publisher
build
OpenAI's confirmed NVIDIA footprint is a rack, not a chip; Rubin is still a roadmap1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One team's word, checkable only against itself
The 81%, the 71%, the 15% escalation rate and the 60% schema share are all one anonymous team describing its own dashboard, with no repository, no traces and no second party anywhere in this story. What could be checked, we checked, and the author earns some credit: the $0.0014 cheap call falls straight out of the $0.20 and $1.20 prices quoted higher up. The fraying is at the top — the headline pair only holds under one reading of "81%" — and a twenty-character prompt described as eleven characters suggests figures typed from memory rather than copied from a report.
One production router, one price sheet
Two concrete things have happened: a cheap-tier price cut the post dates to July 30 with Azure following on August 1, and one team running this router in front of its gateway at a 15% escalation rate. Nobody else in this story reports shipping the pattern, no vendor confirms the prices, and the code that would let others copy it isn't published. That is an existence proof, not a trend.
The headline oversells; the body mostly doesn't
The title credits forty lines of Go. The post's own arithmetic credits a ten-to-one price gap and a request mix that is largely three-word document titles — narrow the spread or harden the workload and the same code returns much less. What keeps the overstatement local is how candid the rest is: the author volunteers four failures, states plainly that latency is now the bill, and takes apart the confidence-score gate he could easily have sold instead.
Reputation, not vendor money
No sponsor, no product, nothing on sale — what this post buys is engineering credibility on a platform where a percentage in the title is the distribution strategy. The incentive shows in the asymmetry of precision: the flattering figures come to four decimal places, while the expensive model goes unnamed, absolute spend never appears, and the latency section the author promises is the one that's missing.
Trust the pattern, not the percentages
The design and the reasoning are on the page in code and can be argued with, which is why we are comfortable describing them. The magnitudes are another matter: one publisher with a stake in the result, a truncated argument, and a headline that resolves only under a favourable reading leave us unwilling to quote 71% as anything but a claim.