Paging the KV cache into 16-token blocks caps the waste at 960 KB per sequence

Every token in flight costs 64 KB of VRAM at the model shape the g factor engineering blog uses. The blog gives the cache size as twice the layer count, times KV heads, times head dimension, times sequence length, times bytes per value ^[7]. With 16 full-attention layers, 4 KV heads and a head dimension of 256 in BF16 ^[8], that comes to 2 x 16 x 4 x 256 x 2, or 65,536 bytes per token ^[10]. At 8,192 tokens the product is 536,870,912 bytes, the half gigabyte the article reports for a single sequence ^[11][9].

The fragmentation example is worth converting into the same units. The blog reports 15 GB of free VRAM and a crash on the next request needing a contiguous 2 GB block ^[12]. At 64 KB a token, that failing 2 GB is four 8,192-token sessions ^[18]. The 15 GB sitting free is thirty of them, seven and a half times what the failing request asked for ^[19]. None of it is in one piece, so the request dies with a CUDA out-of-memory error ^[12].

Paging bounds the waste instead of removing it. A 16-token block holds 1 MB at this model shape, a 32-token block 2 MB ^[20]. Each sequence can leave one block partly filled, so the worst case is 15 empty token slots, about 960 KB, or 31 slots and about 1.9 MB with the larger block ^[21]. Set that against reserving each slot's worst-case length up front: 8,192 tokens of headroom you reserve and never use is 512 MB per concurrent slot ^[11][14]. The article dates the underlying trick to 1960s operating systems, 4 KB pages and a page table onto scattered physical memory ^[15].

In the blog's framing, text generation is a memory bandwidth problem disguised as a compute problem ^[2], and the decode description supports that: bandwidth-bound matrix-vector work ^[5], with weights and cache shuttled across HBM for every emitted word ^[3]. Its own list of what breaks in a naive PyTorch stack is memory fragmentation, static-batching bubbles and host-side driver overhead ^[6]. Two of those three are allocation and scheduling. Neither fix adds HBM bandwidth: PagedAttention changes where the cache lives ^[14], and continuous batching changes when a waiting request is admitted, at the iteration some sequence finishes ^[16].

The 12% utilization figure is a claim about someone else's traffic. It appears as something an engineer sees in nvidia-smi during a production run ^[1], on the dedicated H100 and H200 clusters the blog says it used ^[17]. For it to describe your service, decode has to dominate your token mix, your effective batch has to be small, and requests have to be queued behind sequences that have not finished. Long prompts with short answers put the time in prefill instead, which the article describes as dense, compute-heavy matrix multiplication ^[4].