MPS buys ASR throughput until the p99 crosses 1,000ms

Time-slicing and MPS both land several processes on one device, and the difference is where the work meets the scheduler. The writeup's own definitions: time-slicing interleaves work from multiple processes on the same device, while MPS lets multiple CUDA clients run concurrently through a single GPU context ^[3]. Concurrent execution in the SMs is the point, while interleaving just takes turns.

The adoption cost is why this is a candidate at all. NVIDIA CUDA MPS is described as a binary-compatible alternative implementation of the CUDA API, which is what lets existing CUDA workloads pick up concurrent execution without being rewritten around a different programming model ^[4]. You are changing how the server process is launched, not the graph it executes. That is a config change and a restart, which is a cheaper week than a distillation run.

Now the arithmetic, because the headline figure is a claim about someone else's request mix. Cost per request is instance price per hour divided by requests served per hour. Hold the instance fixed, and a 75% cost cut ^[2] requires four times the requests per GPU-hour: 1/(1-0.75) = 4 ^[14]. So four clients' worth of concurrent ASR work has to fit in 48 GB of device memory and still keep mean latency under 650 ms with p99 under 1,000 ms ^[10]. That envelope permits a tail 1.54x above the mean ^[15]. A tighter ratio than that and you stall at a lower client count, and you bank a fraction of the 75%.

The tuning method is sequential, and it is the transferable part ^[12]:

1. Raise concurrency while utilisation improves. 2. Track mean and p99 together rather than throughput alone. 3. Stop at the last level that still satisfies the service objective.

The precondition the authors state is the one to check against your own fleet: that a single model instance per GPU is often not the most efficient use of the hardware at scale ^[13]. If your one instance already saturates the device, there is no idle capacity for a second client to buy.

What the material does not supply is the baseline. One evaluated configuration is named, Triton plus MPS on g6e.4xlarge ^[9], and the text carries no baseline throughput, no concurrency level at the operating point, and no instance price to check the ratio against ^[16]. The reproducibility scaffolding around it is genuinely careful: the serving side is pinned to nvcr.io/nvidia/tritonserver:26.03-py3 ^[7], the model artifact is baked in at build time through Dockerfile.single and the LOCAL_NEMO_FILENAME build argument ^[8], and the pipeline is split into three containerised components so a sharing-mode swap leaves the rest of the flow comparable ^[6].

The claim I would defend from this is narrower than the number. For an ASR service whose model already works, sharing mode is the first knob to reach for, because the cost came from how the GPU was shared rather than from replacing the model or redesigning the pipeline ^[11]. That is a deployment decision with a rollback, while model work is a project with a schedule.