Your Multi-Key Failover Is The Most Expensive Line On Your Coding Agent Bill

A writeup published on dev.to for a gateway called VMR argues that the resilience feature most teams add without thinking, several API keys behind a load balancer, is quietly the largest cost driver in a coding-agent bill [s1c3]. It matters because prompt cache discounts are now big enough that where a request lands determines what it costs: Anthropic, DeepSeek and OpenAI all bill a hit on a repeated input prefix at 10-20% of full price, according to the post [s1c1].

The mechanism is dull and structural. Prompt cache is isolated per upstream endpoint, meaning per account and per physical node [s1c2]. Round-robin request one to account A, request two to account B, request three back to A, and every round looks like a brand-new session to every account, so nothing hits and everything is billed at full rate, including the roughly 90% of tokens already computed in earlier turns [s1c3]. Put the discount tier and the miss side by side and a miss costs five to ten times a hit on the same tokens [s1d1].

The awkward part is that gateway scheduling was built for the opposite goal. Smooth weighted round-robin and greedy quota balancing exist to spread traffic evenly; cache locality wants traffic pinned [s1c5]. In long sessions, fair and cheap are not the same objective.

VMR's answer is an in-memory registry mapping a session fingerprint to the endpoint that last served it [s1c6]. The fingerprint choice is the detail worth stealing: it is a hash of the system prompt plus the first user message, deliberately not the client session ID, because agent frameworks regenerate IDs on restart while the cache prefix is determined by content [s1c7]. Routing then has two layers. A sticky pin outranks all quota scheduling, and the session stays pinned even when that account's quota is exhausted, on the reasoning that switching mid-session zeroes the cache and recomputing tens of rounds costs an order of magnitude more than a short overage [s1c8][s1c9]. Only unbound sessions enter scheduling, assigned greedily by quota slack after priority ties; SWRR is skipped because it needs a persistent accumulator and spreading traffic is itself anti-cache-locality [s1c10].

Two operational notes. TTL is tiered: a 10-minute default, hard-capped at 24 hours by a memory-eviction backstop, with per-endpoint overrides such as two hours for disk-cached providers like DeepSeek whose caches survive hours to days [s1c11][s1c12]. The expiry sweep is event-triggered and throttled rather than a ticker goroutine, to avoid a global lock walk on every call [s1c13].

This puts cache affinity in direct conflict with the other popular cost tactic. Local compression, such as OpenProxy's RTK, assumes the upstream is stateless and bills everything at full price, so it re-encodes tables, injects terse-output instructions and prunes tool schemas, claiming 20-40% savings [s1c14]. Each of those edits invalidates the prefix hash from token zero [s1c16]. The post's arithmetic: in a 30-round session, shaving 30% of tokens does not pay for re-buying 29 rounds of accumulated context at full price, so compression can make you more expensive [s1c17]. The author scopes the criticism to RTK in long sessions and credits OpenProxy's broader feature set, including OAuth subscription pooling for ChatGPT, Codex and Gemini, a web UI, hedging and fusion scheduling, and MCP and multimodal extensions [s1c18].

Treat the headline numbers with the caution the author asks for: the 80-95% hit rate and 50-70% bill reduction come from the project's own 150 req/s load tests, self-observed and not third-party audited [s1c4].

What to watch: whether your provider documents cache scope per account rather than per organisation, since the whole argument rests on that isolation [s1c2]; whether anyone reproduces the 50-70% figure on a bill they did not build the router for [s1c4]; and whether compression vendors start publishing cache-hit rates alongside token-reduction percentages, because a token count is not a price [s1c16].