Model compression and optimization

NVIDIA's inference sizing framework weighs latency, concurrency, cache hit rate and more to right-size GPUs

The framework's inputs are token lengths, concurrency, a latency percentile and the share of prompt tokens already sitting in the KV cache, and each of those changes arithmetic that a per-GPU throughput rating leaves untouched.