Build1 publisher3 min readPublished
Confidential inference on Blackwell retains 96-98 percent of throughput with CC-aware adaptations, NVIDIA reports
NVIDIA measured TensorRT LLM holding 96.1 to 98.2 percent of its non-confidential output throughput on Blackwell, and it got there by unpinning host memory on the affected paths, moving decode readback off the scheduler thread, and timing kernel tactics with the GPU's global timer instead of CUDA events.
The Engineer · Build desk
What happened
- NVIDIA reports TensorRT LLM retaining 96.1 to 98.2 percent of its non-confidential output-token throughput across concurrency 1 to 16, with mean time per output token within 1.2 to 4.3 percent of baseline.
- TensorRT LLM now chooses pageable memory instead of pinned memory on the affected transfer paths and, in PR #11573, moves repeated token and sampling-data readback to an asynchronous worker.
- In the tested confidential configuration the autotuner's CUDA-event timestamps were unstable enough to select a slower tactic, so TensorRT LLM times candidate tactics with the GPU %globaltimer.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A retention percentage for confidential inference is only legible next to a framework version, because this one was measured on a runtime carrying three CC-specific changes to allocation, threading and timing.
- decision Allocation policy becomes a runtime check: the pinned-memory call that is the fast path on a normal host is a blocking path inside a CVM, so serving code has to branch on CC state.
- exposure Anything that times GPU work with CUDA events inside a CVM can get a wrong answer without raising an error. Custom kernel selection and in-house profiling built on the same primitive sit on that timer too.
- capability The two published formulas let a platform team produce its own retained-throughput and TPOT numbers at its own concurrency instead of inheriting a band measured between 1 and 16.
The bounce buffer is where an unadapted stack pays first. On B200 with confidential computing on, the GPU cannot directly access protected CVM memory, so host-to-device transfers pass through a software encrypted bounce buffer [11]. Pinned host memory stops delivering its usual asynchronous-transfer advantage, and some copies block the calling thread [12]. Pinning the host buffer you copy from is the first optimization every CUDA guide teaches, and on these paths it is the one to take back out. TensorRT LLM's change is CC-aware memory selection: pageable memory on the affected paths instead of pinned memory unconditionally [13].
Decode is where the second cost lands. Token and sampling data are read back to the host on a repeating basis, and under CC those protected copies would block the main scheduler. TensorRT LLM moved the readback to an asynchronous worker, in PR #11573 [14].
Kernel selection broke differently. The autotuner compares candidate tactics using CUDA events [15]. In the tested CC configuration the event timestamps produced an unstable timing signal, and the autotuner could select a slower tactic on the strength of it [16]. It now measures tactics with the GPU `%globaltimer` [17].
Invert the worst reported latency overhead and the two headline numbers converge. 100 divided by 1.043 is 95.9 percent, within 0.2 points of the worst retained-throughput figure of 96.1 percent [4][5][2]. The worst measured throughput loss is 3.9 points [1]. At concurrency in the single digits that is what you would expect: throughput per request is the reciprocal of time per output token, so both numbers are measuring the same decode-path stall.
NVIDIA selected the workload to make that stall visible. Long input context stresses data movement during prefill, extended generation amplifies small per-token CC costs during decode, and low concurrency limits the chance to hide those costs across concurrent requests [8]. High request volume does the opposite, amortizing fixed encryption costs by overlapping stalls with other work and making the direct effects harder to observe [9]. For a workload of this shape, the 1-to-16 band should sit near the maximum visible overhead, and adding concurrency hides more of it [5].
Before the band transfers to your deployment, your runtime has to select memory with CC state in mind, keep the repeated decode readback off the scheduler thread, and time candidate tactics with something other than CUDA events [13][14][17]. Your workload has to match the one in the post's Table 2 configuration, or beat it on concurrency [8][18]. NVIDIA states that secure execution changes the assumptions behind memory movement, timing, scheduling and multi-GPU communication, and that overhead follows if the runtime does not adapt [2]. The figures in the post cover the adapted TensorRT LLM, and NVIDIA did not publish a number for a runtime without those changes [3].
What to watch
- Whether anyone publishes the same CC-on versus CC-off comparison for a serving runtime without the three CC-aware adaptations.
- Whether measurements appear above concurrency 16, where NVIDIA says fixed encryption costs get amortized across requests.
- Whether the encrypted NVLink path gets its own measured overhead, since multi-GPU communication is listed among the changed assumptions.