Transformer inference

How transformer models compute at serving time: the prefill pass over a prompt whose tokens are all known, autoregressive decoding that produces one token at a time, and the key-value cache holding intermediate state for tokens already processed.