Cached prefixes push 91% of a ReAct agent's LLM time into decode

A ReAct turn arrives at the server looking almost exactly like the previous one. Same system block, same tool schemas, same transcript of earlier thoughts and observations, with one tool result appended at the end. Prefix caching keeps the KV blocks up to the point where the two prompts diverge, and that point sits near the very end. Prefill only pays for that tail, while generation carries the rest of the work, and that split is why the traced runs put nearly all LLM time in decode ^[3]. The paper's framing is that this is a different animal from conventional long-prompt serving ^[13], and the mechanism backs that up: the long prompt is already in memory.

The arithmetic across the two ends of the measured hit range is worth doing, using SWE-bench Pro's average accumulated context as the stand-in ^[7][8]. At a 99.5% hit ratio, an 80K-token prompt needs roughly 400 tokens of fresh prefill per turn ^[15]. At 84.6%, the same prompt needs about 12,300 ^[16]. That is thirty times the prefill work for an identical context, decided by whether the blocks were still resident ^[17]. A fully cold prefix is worse, because the whole 80K gets recomputed, 200 times the best case ^[18]. Mixing a per-benchmark context size with an across-configuration hit rate makes that illustrative rather than measured, but the shape holds at whatever context size you substitute.

For the numbers to transfer to your stack, the prompt prefix has to be byte-stable turn over turn. A clock in the system prompt will do it. So will a tool list serialized in map iteration order, or a retrieval block prepended above the transcript, because each moves the divergence point to the top and takes the hit ratio with it. The state also has to survive the gap while the tool runs, which is dead time the blocks spend holding memory without producing tokens.

The paper flags the practical limit itself. Achieving decode-dominated execution is hard because concurrent agents with long multi-turn contexts can exceed GPU capacity, and the sentence in the copy I have stops at "can exceed GPU" ^[5], so I will not put a number on the ceiling.

Re-entry count multiplies the exposure. On ADE-Bench, Gemma Thinking averaged 18.0 turns while Gemma Instant averaged 108.8, with one trajectory reaching 786 ^[9]. Six times the turns is six times the number of moments the session's state has to still be there when the model comes back ^[19]. A 786-turn trajectory is something only a benchmark harness would sit through.

The instrumentation that follows from this is per-session rather than per-fleet: prefix hit ratio, and the interval between an eviction and the next re-entry that needs the evicted blocks. Aggregate tokens per second will look healthy while individual agents quietly re-prefill their entire history.