Build1 publisher2 min readPublished
A 100K-token agent session prefills 3M tokens over thirty turns without prefix reuse
DigitalOcean's serving guide puts the cost of a long request on two mechanics, the prefill pass and the KV cache, and says the real choice is which mix of caching, prefix reuse and retrieval fits the budget.
The Engineer · Build desk
What happened
- A DigitalOcean serving guide argues that accepting a million tokens and serving a million tokens to real, concurrent users inside a latency and cost budget are two different engineering problems.
- It puts the speed of a long request on two mechanics: the prefill pass that reads the whole input before anything is written, and the KV cache that stores every token's key and value vectors.
- Its worked example is a coding agent or a multi-turn document review holding state across dozens of turns without resending the same 100K-token context on every message.
- The recommended stack is KV cache economics plus prefix reuse such as SGLang's RadixAttention, DigitalOcean's Inference Router for cache-aware routing, and retrieval-augmented generation alongside.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost A team that ships a long window without prefix reuse pays for the same prefix on every turn, so the invoice grows with context length while the questions stay the same size.
- decision Anyone quoting an interactive response time on a million-token endpoint has to settle the caching and routing arrangement first, because the window number will not hold the latency target on its own.
- constraint A time-to-first-token commitment has to be quoted per context length, since the first token waits on the prefill pass over whatever the caller sent.
- capability When the prefix stays resident between messages, multi-turn document review and coding agents become affordable to run with 100K tokens of state.
Put numbers on the guide's own example. A coding agent carrying 100,000 tokens of context across thirty turns resends that prefix on every message unless the serving layer reuses it [5]. Thirty turns times 100,000 tokens is 3,000,000 tokens of prefill for one session [13]. With prefix reuse, the 100,000 tokens are read once and each turn pays only for its increment. The post gives token counts; it does not give figures for memory per token, time-to-first-token or price [14]. The multiplier is the part that transfers to your cluster.
Nothing streams until the prefill pass over the whole input finishes [3]. DigitalOcean's guide argues that understanding the prefill and decode split is how a team sets a time-to-first-token target it can actually hit [15]. It names prefill/decode disaggregation as the two-phase structure behind long-context serving decisions [10]. It also argues that an advertised window describes what the model reads in isolation. Time-to-first-token, batch size and the GPU bill when dozens of those requests land on one cluster at once are a separate question [4].
The check it specifies for a serving layer is narrow and testable: does the layer reuse the KV cache across requests that share a common prefix, or recompute it each time [9]. I would answer that before believing any window number. Send the same long prefix twice and compare time-to-first-token.
Two of the named remedies are generic and one is a product. KV cache economics and prefix reuse stand on their own, with SGLang's RadixAttention named as an implementation; cache-aware routing arrives as DigitalOcean's Inference Router [6], in a post published under DigitalOcean's account on dev.to [11]. The comparison the guide reaches for is ten novels of text at a million tokens [1].
The claim with the most consequence for design is about accuracy. Models answer less reliably as input grows, according to the post, which advises sending precision lookups to retrieval instead of trusting long-context recall, and testing accuracy at the context lengths you will actually run [7]. The post is citing evaluations it did not run. For it to hold on your workload, your retriever has to find the right passage at least as often as the model recalls it from a full window. Measure that on your documents, at the length you plan to serve. Where that holds, the guide treats retrieval as both the alternative to a long window and its complement [6].
What to watch
- Whether DigitalOcean publishes the memory, time-to-first-token and price figures its takeaways promise.
- Whether Inference Router's documentation states a prefix cache hit rate or how long a prefix stays resident.
- Accuracy measured at 128K and at 1M on the same task, from any provider, so the long-context recall claim can be checked.