Tail latency and service objectives

MPS buys ASR throughput until the p99 crosses 1,000ms

AWS, NVIDIA and Heidi report cutting production speech-recognition inference cost by 75% by sharing one GPU through CUDA MPS instead of running a single model instance. How much of that saving transfers depends on the latency envelope you accept.