NVIDIA's 5x first-token figure needs the vision encoder to own four fifths of your TTFT

In aggregated serving, one worker owns vision encoding, LLM prefill and decode inside a single request lifecycle ^[6]. A request carrying several images therefore runs its own vision transformer pass ahead of its own prefill on the same GPU, and that pass can take hundreds of milliseconds or longer while contending with other requests' prefill and decode ^[7]. Encode-prefill-decode disaggregation breaks that coupling: the encoder becomes its own worker with its own queue and batching, and the resulting embeddings cross to the prefill-decode worker through NIXL ^[1][9][11].

Four fifths of baseline time to first token has to come off the critical path for the headline speedup to appear. The arithmetic is not subtle: a 5x reduction leaves the new TTFT at one fifth of the old, so 1 - 1/5 = 0.8 of the original latency must be work the split removes or overlaps ^[1]. Invert it and you get the transfer test for your own traffic. If encode and its queueing are 20% of your TTFT, the ceiling is 1/(1 - 0.2) = 1.25x before you pay anything for coordination ^[4], and NVIDIA is explicit that the isolated encode work has to be large enough to offset worker coordination and embedding transfer overhead in the first place ^[10].

The stated scope is narrow: image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models ^[2]. The material gives that last condition without a mechanism, so it reads as an observation from NVIDIA's own runs rather than something you can reason forward from.

The negative cases are the useful part of the post. Aggregated serving stays the right design when media processing is only a small portion of the workload ^[6]. Disaggregating the encoder onto same-class GPUs is worse than colocation every time, which is why NVIDIA drops that configuration from its comparison entirely ^[15]. The reason is sizing: the vision encoder is lightweight next to the LLM, so a whole same-class GPU dedicated to encode work sits largely idle ^[12]. Reserving a top-tier accelerator to run a ViT is an expensive way to keep a device warm.

That leaves colocation as the default on a homogeneous fleet ^[12], and the separate-tier topology as something you can only buy your way into. NVIDIA's disaggregated test ran one encoder GPU for every two prefill-decode GPUs ^[14][2], with the encoders on the cheaper class ^[13]. Note also what the material does not say: the up-to-5x figure is never attributed to that specific topology or to a named workload ^[3].

Hardware availability decides where the encoder can run, and workload characteristics decide whether moving it pays ^[16]. Media volume alone does not settle the second question, because what matters is the share of request processing time vision encoding actually consumes ^[17]. The measurement that decides it is one you take on production concurrency, including the text-only requests currently sitting behind image requests that need no encoder at all ^[8].