Routing on the prompt's first tokens cut AWS's median time to first token by up to 77 percent

Convert the cache numbers into misses, because misses are what the prefill hardware sees. Under random routing three requests in four recomputed the prefix from scratch; with affinity fewer than one in five do, a 76 percent cut in redundant prefill ^[15]. The median time-to-first-token reduction AWS reports on that same run, 71 to 77 percent, tracks it closely ^[7].

Fleet size sets the ceiling on that. Under random routing a request reaches the instance that last served its prefix roughly one time in N: about 14 percent across the seven instances AWS used, and 50 percent across two ^[16]. The smaller the fleet, the less redundant prefill there is to remove, and this benchmark ran on seven.

The tail moved less than the median. P90 TTFT fell 33 to 37 percent against P50's 71 to 77 percent, so the median improved about twice as much as the tail ^[17]. Overload protection accounts for part of that by design: when the instance holding a popular prefix reaches the concurrency limit, the endpoint sends the request to a less busy instance and eats the miss ^[11]. You configure that limit, so you also configure how often the endpoint throws away a hit.

For the long-context figures to transfer, you need an 8,000-token prefix genuinely shared across requests and sustained for an hour, vLLM with prefix caching switched on, and something like Llama 3.1 70B Instruct on p5 hardware ^[5][7]. AWS published its own control case. On variable-length ShareGPT-style conversations the throughput gain came out about eight times smaller than on the long-context run ^[18], and AWS attributes the gap to prefix length, since a longer shared prefix means more computation skipped per hit ^[14].

So the number that predicts your result is the share of prompt tokens that is identical across requests. In the worked example AWS uses, a 3,000-token instruction block sits above a 50-token customer question, which puts 98 percent of the prompt in the shared part ^[19]. Few production prompts are that lopsided once conversation history and retrieved documents start varying per user.

Routing keeps the whole prefix and pays for it once per instance instead of once per request. Whether that beats deleting tokens is not something this benchmark measures: every latency figure in the post is P50 or P90 time to first token, with no end-to-end latency and no comparison against prompt compression or trimming ^[20]. The post also does not say how many leading tokens of the payload the endpoint reads to pick an instance ^[21].