Topic

Inference disaggregation

Serving architecture that runs the stages of a model request (media encoding, prefill, decode) as separate workers that batch, schedule and scale independently, exchanging intermediate state over an interconnect.

Current clusters

build1 publisher

NVIDIA's 5x first-token figure needs the vision encoder to own four fifths of your TTFT

Dynamo can now run the vision encoder as its own worker, and the win is real where encoding dominates a request. The arithmetic behind the headline number tells you which traffic it transfers to, and NVIDIA names the one topology it never beats.

Publishers:developer.nvidia.com

Reality

Evidence62
Adoption20
Hype gap+35
Incentives80
Confidence55