NVIDIA Triton Inference Server

Four concurrent MPS processes fill the L40S that one ASR request leaves 80% idle

AWS, NVIDIA and Heidi Health report holding sub-second transcription while cutting 16 GPU instances to four. Per-GPU throughput rose only about 1.5x, so the rest of that saving came out of provisioning headroom.