NVIDIA's 2.5x concurrency figure rests on a 64K prompt reused 76 percent of the time

Take the reuse figure first. If 64K means 65,536 tokens, then at 76 percent KV reuse about 15,700 tokens of each request go through prefill and about 49,800 come out of cache ^[3][2]. In an agentic loop that is where the headroom is, and it is also the most fragile assumption in the setup. NVIDIA's reuse layer is three things: prefix caching, partial-prefix matching that recovers reuse when only part of a prefix matches, and Mamba state-cache settings tuned for the architecture ^[8].

The interactivity target is stated two ways and they agree: 50 tokens per second per user, and a 20 ms inter-token latency ^[3]. One second divided by 20 ms is 50 ^[1]. At that rate a 400-token answer streams for eight seconds ^[3]. So the scheduler holds each sequence open for eight seconds while deciding how much more to admit, and the knobs NVIDIA tuned there are concurrent-sequence limits, batched-token limits, block size and GPU memory allocation ^[9].

On whether the layers can be summed, NVIDIA writes that "the measured gains come from interacting configuration bundles, not independent switches whose percentages can simply be added" ^[6]. Take that literally. No per-layer split is published, so you cannot lift prefix caching into a runtime that lacks the autotuned mixture-of-experts and Mamba kernels and claim a fixed fraction of the 2.5x ^[5]. MTP speculative decoding is the clearest case: the post says its incremental benefit depends on acceptance rate and available memory headroom, and gives no acceptance rate for the agentic trace ^[10].

The comparison itself is labelled NIM Off against the fully optimized NIM 2.0.12 stack, and NIM Off is described only as the open-source baseline serving stack ^[2]. The post gives no configuration for that baseline ^[15]. So the multiple is measured against a reference point the reader cannot inspect. For it to transfer you would need prompts near 64K, reuse near 76 percent, answers near 400 tokens, a 20 ms target, and four B200s with tensor parallelism spreading the model across all four ^[3][7]. Because one replica spans four GPUs, the smallest capacity increment is another four B200s ^[5].

The part worth keeping regardless of the multiple is the method. NVIDIA tells you to pin the image tag or digest for every run, capture a Mooncake-format JSONL trace of your own traffic with sanitization and access controls, replay it with AIPerf, and select the Pareto point that satisfies your SLO ^[12]. A vendor publishing the harness that can contradict its own chart is doing the useful half of the work. The example loop sweeps concurrency at 1, 4, 8, 16, 32 and 64 against nvidia/nemotron-3-ultra-550b-a55b, so a target above 64 is one you extend the loop to reach ^[13].