Loading today’s stories
Build1 distinct publisher3 min readUpdated
Prompt cache is scoped per upstream endpoint, so round-robin routing turns every agent turn into a full-price cache miss. One gateway writeup puts the sticky-routing saving at 50-70%.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A writeup published on dev.to for a gateway called VMR argues that the resilience feature most teams add without thinking, several API keys behind a load balancer, is quietly the largest cost driver in a coding-agent bill [s1c3]. It matters because prompt cache discounts are now big enough that where a request lands determines what it costs: Anthropic, DeepSeek and OpenAI all bill a hit on a repeated input prefix at 10-20% of full price, according to the post [s1c1].
The mechanism is dull and structural. Prompt cache is isolated per upstream endpoint, meaning per account and per physical node [s1c2]. Round-robin request one to account A, request two to account B, request three back to A, and every round looks like a brand-new session to every account, so nothing hits and everything is billed at full rate, including the roughly 90% of tokens already computed in earlier turns [s1c3]. Put the discount tier and the miss side by side and a miss costs five to ten times a hit on the same tokens [s1d1].
The awkward part is that gateway scheduling was built for the opposite goal. Smooth weighted round-robin and greedy quota balancing exist to spread traffic evenly; cache locality wants traffic pinned [s1c5]. In long sessions, fair and cheap are not the same objective.
VMR's answer is an in-memory registry mapping a session fingerprint to the endpoint that last served it [s1c6]. The fingerprint choice is the detail worth stealing: it is a hash of the system prompt plus the first user message, deliberately not the client session ID, because agent frameworks regenerate IDs on restart while the cache prefix is determined by content [s1c7]. Routing then has two layers. A sticky pin outranks all quota scheduling, and the session stays pinned even when that account's quota is exhausted, on the reasoning that switching mid-session zeroes the cache and recomputing tens of rounds costs an order of magnitude more than a short overage [s1c8][s1c9]. Only unbound sessions enter scheduling, assigned greedily by quota slack after priority ties; SWRR is skipped because it needs a persistent accumulator and spreading traffic is itself anti-cache-locality [s1c10].
Two operational notes. TTL is tiered: a 10-minute default, hard-capped at 24 hours by a memory-eviction backstop, with per-endpoint overrides such as two hours for disk-cached providers like DeepSeek whose caches survive hours to days [s1c11][s1c12]. The expiry sweep is event-triggered and throttled rather than a ticker goroutine, to avoid a global lock walk on every call [s1c13].
This puts cache affinity in direct conflict with the other popular cost tactic. Local compression, such as OpenProxy's RTK, assumes the upstream is stateless and bills everything at full price, so it re-encodes tables, injects terse-output instructions and prunes tool schemas, claiming 20-40% savings [s1c14]. Each of those edits invalidates the prefix hash from token zero [s1c16]. The post's arithmetic: in a 30-round session, shaving 30% of tokens does not pay for re-buying 29 rounds of accumulated context at full price, so compression can make you more expensive [s1c17]. The author scopes the criticism to RTK in long sessions and credits OpenProxy's broader feature set, including OAuth subscription pooling for ChatGPT, Codex and Gemini, a web UI, hedging and fusion scheduling, and MCP and multimodal extensions [s1c18].
Treat the headline numbers with the caution the author asks for: the 80-95% hit rate and 50-70% bill reduction come from the project's own 150 req/s load tests, self-observed and not third-party audited [s1c4].
What to watch: whether your provider documents cache scope per account rather than per organisation, since the whole argument rests on that isolation [s1c2]; whether anyone reproduces the 50-70% figure on a bill they did not build the router for [s1c4]; and whether compression vendors start publishing cache-hit rates alongside token-reduction percentages, because a token count is not a price [s1c16].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
When a load balancer round-robins request 1 to account A, request 2 to account B and request 3 back to A, every round looks like a brand-new session to every account, nothing hits, and everything is billed at full price including the 90% of tokens already computed in earlier rounds.
The global sticky_ttl defaults to 10 minutes, suitable for Anthropic and OpenAI memory caches, and is hard-capped at 24 hours by a memory-eviction backstop so the registry cannot grow unbounded.
Disk-cached providers such as DeepSeek keep caches for hours to days, so sticky_ttl can be overridden per endpoint, for example sticky_ttl: 2h.
Modern providers including Anthropic, DeepSeek and OpenAI charge 10-20% of full price for a cache hit on a repeated input prefix.
Prompt cache is isolated per upstream endpoint: per account, per physical node.
Classic gateway scheduling algorithms such as smooth weighted round-robin and greedy quota balancing are designed to spread traffic evenly, while cache locality requires traffic to be pinned; in long sessions fair and cheap are mutually exclusive.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Mechanism well-specified, results unverified
The causal chain is documented at implementation level by a primary participant - per-endpoint cache scoping, fingerprint hashing, two-layer routing, TTL tiering, a named source file and a working config - which is coherent and checkable in principle. But the cluster holds exactly one source, that source is the project's own author, the only quantitative outcome is self-observed and explicitly unaudited, the request trace is labelled illustrative, and the competitor characterization has no corroborating primary source.
No third-party adoption signal
Nothing in the cluster shows anyone outside the project running this: no release or version announcement, no download, star, user or deployment counts, no customer or team disclosure. The two observable artifacts are the author's own load test and the documented configuration surface, neither of which measures uptake.
Headline savings outrun the evidence
The mechanism - per-endpoint cache scoping punished by even-spread routing - is plausible and internally well-argued, so the story is not empty. The overshoot is in the framing: a 50-70% saving headline rests on one unaudited internal test at a single load point; the sweeping conclusion that compression 'can make you more expensive' is arithmetic on a hypothetical 30-round session rather than a measured comparison; and costs of the fix (pin held through quota exhaustion, concentration on one endpoint, hedging rejected) are asserted as clearly favorable without numbers.
Author is the project's promoter and a rival's critic
The single source is written by a participant in the VMR project, publishes VMR's own unaudited savings figures, ships VMR configuration as the remedy, and simultaneously frames a competing project's compression feature as potentially cost-increasing. Both the problem framing and the comparative verdict come from the party whose tool the reader is being pointed at, though the piece does disclose the numbers are self-observed and credits OpenProxy's broader feature set.
Low - single interested source
One publisher, one author, one interest. Implementation and design claims are credible as first-party description and are internally consistent; the quantitative and comparative claims cannot be checked from the supplied material, and no independent publisher, provider documentation, or rival response is present to corroborate or contest them.
product
A 2x LLM bill is not a bug report: token spend is an observability problem1 distinct publisher
build
A 12MB Go binary bets agent cost control is cache stickiness, not a dashboard1 distinct publisher
leadership
Serval's Catalyst mines the ticket queue for automation work, not the project backlog1 distinct publisher
build
Claude Code now outruns Copilot roughly two to one in JetBrains' survey of 15,000 developers1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 18, 2026