MLA's 512-Scalar KV Latent Cuts Cache 98%, Enabling 512 Concurrent 128k Streams

Divide the 97.8 ms by the eight streams and each concurrent user costs 12.2 ms of every token step ^[15]. Add a ninth and all nine get slower. The 12.2 ms is independent of the model's FLOPS: it is the time to pull one user's history out of HBM3, once, per generated token.

The weights compete for the same bandwidth and they scale differently. Llama 3 70B in FP16 is about 140 GB across tensor-parallel ranks ^[5], and that read happens once per step whatever the batch size. KV traffic scales with the batch. Four concurrent 128k streams come to 163.84 GB, more than the weights ^[16]. Serve one user with a short prompt and the cache is a rounding error against 140 GB, so compressing it 64-fold lands below what you can measure.

The two figures invite a multiplication: one sixty-fourth of the bytes inside the same 97.8 ms budget would be 512 concurrent streams ^[17]. For that to transfer to your deployment, your model would have to be the one MLA is described on. That means a 5120-dimensional hidden state, 128 heads of dimension 128, and a latent of 512 ^[8]. The 40.96 GB per stream belongs to Llama 3 70B ^[5]. MLA's baseline of 32,768 scalars per token is full multi-head attention, one key and one value for each of 128 heads ^[9]. The cache formula keeps the number of KV heads as its own term ^[3]. A model already sharing KV heads across query heads sits well below that baseline before MLA enters, and has proportionally less to gain.

Multi-query attention is the cheap version of the same idea, one KV head for every query head, and the article reports score drops of 3.8% to 6.2% on GSM8k and multi-document recall ^[7]. MLA keeps 128 distinct up-projection slices reading one shared latent, so each head still expresses its own pattern over the 512-dimensional space ^[11].

The RoPE handling is the good engineering here. Rotary embeddings multiply each key by a position-dependent rotation, and that rotation does not commute with the up-projection ^[12]. Store only the latent and the runtime has to up-project and re-rotate every earlier token's key at every decoding step. That is the recompute the cache exists to avoid, and its cost grows with context length ^[12]. DeepSeek's answer is to split keys and queries into a content stream at 128 dimensions per head and a separate positional stream ^[13]. The 1.56% figure counts the content latent alone ^[10]. A positional key is cached too, and the excerpt breaks off before naming its size. The 93% in the headline and the 98.44% in the body describe different caches ^[14].