Chip Huyen puts inference at 10 to 100 times a model's training compute

Take the far end of the estimate first. If serving a model burns 100 units of compute for every one spent training it, a 10 percent cut in serving cost saves ten units, ten times the whole training run ^[19]. At the 1:10 end, the same 10 percent still saves the equivalent of one training run ^[19]. The recap gives the range as Huyen's own reckoning over a model's life and does not cite a measurement behind it ^[23].

Her end-to-end latency formula is time to first token, plus time per output token multiplied by the number of output tokens minus one ^[7]. Put her own example targets through it: 200 ms to first token and 100 ms per output token ^[9]. A 500-token answer then takes 200 ms plus 499 x 100 ms, which is 50.1 seconds, and the first token accounts for 0.4 percent of the wait ^[20]. The other 49.9 seconds sit in the per-token path ^[20].

Reasoning models make the first metric ambiguous, because some generated tokens never reach the user ^[5]. "The first generated token might not be the same as the first visible token," Huyen said ^[8]. Some teams measure time to publish instead, meaning the wait until the user sees anything ^[24].

Goodput counts only the requests that met the targets, and in her example three of ten did ^[9]. Goodput was 30 percent while throughput read ten requests a minute ^[21].

On levers, Huyen worked at Nvidia and left hardware out of the talk. "Even though I find it to be an intellectually interesting topic, it's not relevant to a lot of people because we don't have the power to change the hardware itself," she said ^[11]. She also declined to spend time on replica parallelism, calling it costly and quick to get complicated once you are distributing models of varying sizes across a mix of 80GB, 48GB and 24GB machines ^[12].

Access to the weights decides which set of techniques is open to a team. "If you want to host the models yourself, or if you have access to the model weights, or if you train a model yourself, or you want to fine-tune or distill a model, then model optimizations might be for you," Huyen said ^[13]. Quantization from 32-bit to 8-bit takes weight storage from four bytes per parameter down to one, a 4x reduction ^[14][22]. Her speed argument runs on bit width: "If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it's 8 bits, you only have to do it eight times," she said ^[15]. The 4x storage saving transfers to any deployment that is memory-bound; the speedup transfers only if your runtime executes 8-bit kernels for the operators your model spends its time in. The tradeoff she names is a small quality hit ^[16].