The definition of Tavg does most of the work here. Each request in the sweep generated 1,024 output tokens [6], so the 256-token case spends the run growing from 256 to 1,280 tokens of history, averaging about 768, while the 3,072-token case grows from 3,072 to 4,096, averaging about 3,584 [3]. Measured against average history rather than prompt length, the KV term rose 4.7x, not 12x [3].
Time per output token went from 2.998 ms to 3.407 ms, a rise of 0.41 ms [2]. If per-token time tracks per-token bytes at roughly fixed bandwidth, then 4.7x on the KV term buying 13.6% more time puts the KV read near 4% of per-token traffic at 768 tokens of average history [4]. The rest is W. At batch 1 the engine reads the whole weight set to produce a single token, because the decode input is one row per active request: X is [B, H] and each linear layer is [B, H] @ [H, D] [11]. The history it attends to is the paged KV state that BlockManager tracks and ModelRunner feeds to the model [13].
That is where the two knobs separate. W/B is one fixed read divided across the tokens a batch produces, so raising B is nearly free in bytes. KV(Tavg) is charged per request and divides by nothing [3]. Longer contexts raise a floor that continuous batching cannot lower, and prefill sits on the other side of the machine entirely, judged against a local BF16 GEMM reference of 71 to 77 TFLOP/s [15].
What the reported runs establish is narrower than the model. The sweep varies context at batch 1 [5]; the amortization half of W/B comes out of the code and the algebra rather than these measurements. The 531 to 562 GB/s figure is an estimate assembled from computed KV traffic plus the weight term, and the author says it is neither a direct DRAM measurement nor a kernel bottleneck diagnosis [7][8]. The accompanying figure caption is equally careful: compute-heavy prefill and memory-heavy small-batch decode are called workload hypotheses, not labels for every shape [14].
The headline comparison deserves less weight than the sweep. nano-vLLM's official benchmark puts it about 5.3% ahead of vLLM, 1,434 against 1,362 tok/s on 256 requests, on an RTX 4070 Laptop GPU [9][5]. For that to transfer you need the same offline regime where all 256 requests are available at once, the same small model whose weight term collapses quickly as B grows, and a stated vLLM version and configuration, which the benchmark line does not give [10]. A 72 tok/s gap is small enough that an unnamed version difference could account for it [10].
Priced this way, small-batch decode on this card buys weight reads and long-context decode buys KV reads, and only the first of those gets cheaper when you add requests [3][4].
