Build1 publisher3 min readPublished
Dynamo-Triton 26.07 puts eight GPU ranks behind a single named model endpoint
NVIDIA's TensorRT backend now creates per-rank contexts and NCCL communicators inside one Triton instance, and the client sends a single gRPC request to a model name. The GPU count is fixed at compile time.
The Engineer · Build desk
What happened
- NVIDIA Dynamo-Triton 26.07, the release formerly named Triton Inference Server, switches on the multi-device inference capability of its TensorRT backend.
- That path executes a single TensorRT network across multiple GPUs using NCCL-backed distributed collectives, and NVIDIA says it is fully supported starting with TensorRT 11.0.
- One KIND_MODEL instance owns the GPUs, creates a TensorRT execution context, CUDA stream and NCCL communicator per rank, and launches the ranks together for each request.
- The worked example serves the 36-layer denoising transformer of Cosmos 3 Nano, spreading its 44,160 video tokens over as many as eight GPUs using Ulysses context parallelism.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The GPU count is baked into the compiled plan, so a team that wants two, four and eight GPU service levels maintains and versions four engines, and switching level is a deploy.
- decision Whoever operates the serving tier inherits rank and communicator lifecycle; the application team keeps a model name, a gRPC stub and no NCCL code.
- cost Because 6.6% of generation sits outside the transformer, even flawless eight-way scaling lands near 5.5x, so the fifth through eighth GPU buy noticeably less than the first four.
- capability With reduce-scatter, all-to-all and all-gather reachable as public TensorRT layers through converters, a context-parallel graph can be compiled out of a PyTorch export instead of hand-managed in the serving process.
The CP8 model in NVIDIA's post declares one KIND_MODEL instance, `enable_multi_device` set to `"true"`, and `multi_device_gpus` listing GPUs 0 through 7 [12]. Those parameters select a plan. The distributed Ulysses graph is compiled into each context-parallel TensorRT plan before deployment [8], and NVIDIA states that the Dynamo-Triton configuration activates that plan and does not convert a single-device engine into a distributed engine [11]. So the four deployed variants in the example need four compiled plans [25], each exported from PyTorch and compiled with Torch-TensorRT [16]. Moving from two GPUs to eight is a rebuild and a redeploy of a new versioned model.
The return on those GPUs is bounded before any are booked. The transformer is 93.4% of single-GPU generation time [9], leaving 6.6% in the Diffusers stages that stay outside the service: prompts, latents, classifier-free guidance, scheduling, VAE decode, frame postprocessing [6]. Hold that 6.6% fixed and no GPU count takes total generation below roughly a fifteenth of baseline time [21]. Give the transformer flawless eight-way scaling and total generation lands at 18.3% of baseline, about 5.5x [22].
Each accepted plan carries two initial reduce-scatters, three all-to-alls inside every one of the 36 layers, and one final all-gather, a topology of 2 plus 108 plus 1 [17]. That is 111 collectives per forward pass [19]. A generation is 70 forward passes, because each of 35 denoising steps needs an unconditional prediction and a prompt-conditioned one, issued as two sequential Triton calls [10]. Multiply and one video costs 7,770 NCCL collective operations [20]. The calls are sequential by construction, so whatever the gRPC round trip and launch path cost per call, the workflow pays it 70 times.
The sharding is also partial. At CP8 each rank processes 5,520 video tokens outside attention while the shorter 2,992-token text path stays replicated on every rank [14]. Per-rank non-attention token work is therefore 8,512 tokens, 35% of it the duplicated text path [23]. Inside each layer, Ulysses changes the partitioning axis around attention so that every rank processes the full video sequence for a nonoverlapping subset of heads [15].
For the scaling to transfer, a workload has to look like this one: a fixed profile at 44,160 video tokens [7] with a single stage holding 93.4% of wall clock [9]. A pipeline where the accelerated stage is 60% of generation time collects a fraction of the same win from the same eight GPUs. NVIDIA frames the capability as teams trading additional GPU resources for shorter request latency while the application interface stays stable [5]; the post describes all four variants running on one healthy eight-GPU system [18] and stops before the measured latencies [24]. The engineering worth crediting is the boundary itself. The client sends one named model a request and never sees a rank [4], and the per-rank execution contexts, CUDA streams and NCCL communicators live inside a single KIND_MODEL instance [3].
What to watch
- Published CP2, CP4 and CP8 latency numbers for Cosmos 3 Nano would show how much of the 5.5x perfect-scaling ceiling the 111 collectives per pass leave behind.
- Whether the three converters that lower export operations to the distributed-collective layer move from local code into the shipped Torch-TensorRT converter set.
- How a KIND_MODEL instance behaves when one of its eight ranks fails mid-request, and whether Triton surfaces that per rank.