Build1 publisher2 min readPublished
Halving temperature turns a 4:1 token preference into 16:1
Shrijith Venkatramana's walkthrough of sampling puts the odds-ratio arithmetic behind temperature on the page. It shows how much of the difference between two runs of one prompt is settled after the model has finished computing.
The Engineer · Build desk
What happened
- A language model emits a probability distribution over its whole vocabulary, about 50,000 candidates in the article's framing, and every choice made after that point belongs to the decoder.
- Researchers studying GPT-2 in 2019 found that decoding strategy alone could dramatically change the character of generated text while the underlying neural network stayed identical.
- Greedy and beam-style decoding tended toward bland repetition, unrestricted sampling wandered into low-probability nonsense, and carefully truncated sampling came out closest to human writing.
- GPT-2-era generation recipes commonly paired temperature 0.7 with top-k=40, a combination the article cites as the standard of that period.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A team fighting repetition and drift at once cannot fix both from the temperature field, because that dial moves the whole distribution rather than the boundary of the tail.
- decision Pinning a model version leaves output behaviour open. The decoding profile has to be pinned in the same reviewed place before a run is repeatable.
- capability One set of weights can serve a committed profile for extraction and a loose one for brainstorming, so output character becomes a per-task setting instead of a retraining project.
- cost Chasing output variance through prompt rewrites burns review cycles when the cause is a sampling default that arrived with an SDK example.
Temperature acts on ratios. Divide the logits by T before the softmax and a preference of P(A)/P(B) reaches the sampler as approximately (P(A)/P(B))^(1/T) [5][7]. Take a token the model favours 4:1. At T=2 the sampler sees 2:1 [8]; at T=0.5 it sees 16:1 [9]. The ranking stays the same while the odds move by a factor of eight [10]. "It changes how strongly the decoder believes the model's ranking," Venkatramana wrote [11].
For the prompt "The cat sat on the", the article gives mat 0.42, floor 0.20, chair 0.12, bed 0.08, table 0.05 and roof 0.01 [12]. The five named alternatives add up to 0.46, more than mat carries on its own [13]. Greedy decoding takes the argmax [20], so it commits to a token holding less of the listed mass than the candidates it passed over.
Sequence length does the rest. With two reasonable options at every step at 0.8 and 0.2, taking the 0.8 branch twenty times running has probability about 0.8^20, or 0.0115 [14]. So roughly 98.9 percent of runs deviate at least once [15]. An exact-match assertion against that configuration is correct about one time in a hundred.
The long tail is why one dial is not enough. A few tokens are very plausible, and thousands below them are increasingly questionable [16]. Raising temperature spreads probability across all of them and does not distinguish a reasonable alternative from nonsense in the tail [17]. The article frames temperature, top_k, top_p, min_p and repetition_penalty as five answers to a single question: which parts of the distribution you are willing to trust [18].
Nothing here is measured. The distributions are illustrative, and the supplied text breaks off mid-sentence where it credits Holtzman and colleagues with a later result on temperature [21]. The arithmetic transfers to any workload whose next-token distributions look like the example, with one dominant candidate, a few plausible neighbours, and a long tail underneath [16]. Few teams have those distributions for their own prompts, which is the honest limit on the claim that decoding explains their observed variance. In my context that still argues for holding sampling parameters in reviewed configuration, one profile per task. Otherwise the value is never written down, and it cannot be deliberately changed.
What to watch
- The supplied excerpt stops at Holtzman and colleagues, so the equational treatment of top_k, top_p and min_p, and where each one cuts the tail, is still to read.
- Published next-token distributions from a real production workload would show whether the article's example shape is typical enough for its arithmetic to transfer.
- A measured comparison of output variance from model choice against variance from decoding settings would test the framing directly.