Shadow engines cut LLM restart from 283 seconds to 7.3, and change what headroom is for

The interesting part is not the standby process, it is who owns the memory. Normally GPU memory is bound to the engine's CUDA context, which is bound to the engine process, so when that process exits the driver reclaims everything, including weights already resident in HBM, and the replacement has to load them again from storage ^[7]. Dynamo's GPU Memory Service moves those regions into a per-GPU sidecar that is mostly dormant and holds no CUDA context of its own; it allocates physical pages, hands out handles, and arbitrates who may read or write ^[9]. Engines import the handles and map the pages into their own address spaces once, at startup, after which the sidecar is out of the access path ^[10]. Underneath sits the CUDA Virtual Memory Management API, where physical allocations are reference counted and survive as long as any process still maps them ^[11]. NVIDIA's claim is that a read through a GMS-backed pointer costs no more than one through engine-allocated memory ^[12].

Memory sharing only solves half of it. NCCL and torch.distributed communicators bind to a specific running process, and CUDA graphs are pinned to the virtual addresses they were captured at, so that state cannot be handed over and has to be rebuilt on every restart ^[8]. The only way to skip that bill is to have paid it in advance, which is what the idle shadow is for ^[2].

The ratio NVIDIA leads with is nearly 39x ^[5]. The number that matters to a capacity planner is the 275.7 seconds of degraded service the test removed ^[1], and what was happening during them: two workers, one killed, the survivor carrying the entire load at twice its nominal share, with higher TTFT and a slower per-user decode rate for the duration ^[4][2]. Spare capacity for engine loss is priced against that window. If it lasts minutes, the fleet has to be able to serve peak traffic with a worker missing and still meet its latency targets. At 7.3 seconds, which is roughly 2.6% of the cold-restart window ^[3], the same failure looks more like a queue hiccup than a capacity event, and the shadow that absorbs it carries zero marginal weight cost on the GPU it already shares ^[14].

Two limits are worth holding onto. The shadow lives on the same GPUs as the active engine ^[2], and the fault classes named are software ones: process crashes, recoverable CUDA errors, transient collective failures, with node, driver, and hardware still healthy ^[6]. Nothing here survives losing the machine, so the conventional spare-node headroom does not disappear, it just stops being the answer to the most common failure. Second, the only cost NVIDIA puts a figure on is weights ^[3][14]. An idle, fully initialized engine also holds allocations that are not weights, and the post does not price them. This is a preview feature measured on one model and a two-worker topology by the vendor that ships it ^[2][4], with integration described as a narrow change to frameworks including vLLM, SGLang, and TensorRT-LLM ^[15].