Start with the SM budget, because that is what the configuration is actually packing. Transcription instances give each of four concurrent processes a 25 percent SM allocation [17]; diarization instances give each of eight processes 12 percent [18]. That is 100 percent of the card claimed on one and 96 percent on the other [2]. A single request needs 15 to 20 percent of the L40S's 142 streaming multiprocessors [6], roughly 21 to 28 of them [5], so each slice is sized between 1.25x and 1.67x the measured footprint of the work it holds [6].
Memory was never the binding constraint. Four transcription processes at about 2.5 GB each hold 10 GB of a 48 GB card, and eight diarization processes at 1.8 GB hold 14.4 GB [17][18][3]. A 48 GB card storing 10 GB of weights is an expensive filing cabinet.
The throughput figures deserve the same arithmetic. Per GPU, the baseline is about 62 requests per second inside a mean under 650 ms and p99 under 1,000 ms [7]; the MPS and Triton configuration reports 92.1 RPS per GPU at sub-second latency [10]. That is a 1.49x gain [1]. The fleet reduction is 4x, from 16 instances to four [9]. Those do not reconcile through concurrency: 16 GPUs at 62 RPS is 992 RPS of benchmarked capacity, four at 92.1 is 368.4, a ratio of 2.69 [4]. AWS names the difference without quantifying it, describing the 16-GPU deployment as covering peak traffic with sufficient headroom for latency SLAs [8]. Most of the 75 percent, then, comes from re-provisioning against real peak load, with the concurrency gain supplying the margin that makes it survivable.
For the ratio to transfer, three things have to hold in your deployment. Your per-request SM footprint has to be as small as Parakeet TDT 0.6B V2's [6][20]. Your latency budget has to survive contention, since MPS has all clients share one set of GPU scheduling resources rather than rotating exclusive access [14][3]. And your current fleet has to be carrying headroom comparable to Heidi's [8].
The mechanism itself is clean work. MPS is a binary-compatible alternative implementation of the CUDA API, so existing applications run unmodified [11], with all CUDA work funnelled through a single GPU context managed by a daemon [12] and partition size set by CUDA_MPS_ACTIVE_THREAD_PERCENTAGE [15]. What you give up against MIG is physical partitioning with dedicated memory controllers [13]; what you keep is memory protection between clients through separate address spaces [16].
Which is why the title of the post is generous to the daemon. The same solution overview lists request scheduling with Triton and a TensorRT Execution Provider routing ONNX graph nodes for the compute-heavy encoder [19]. The 16-to-four number measures the stack.
