Science1 publisher3 min readPublished
Why LLM output is hard to reproduce: it's not just concurrency and floating point
Thinking Machines Lab argues that temperature-zero inference varies run to run not because GPUs are chaotic but because kernels change their reduction order with load, which makes bit-identical output an engineering target.
The Scientist · Science desk
What happened
- Thinking Machines Lab notes that LLM endpoints stay nondeterministic even at temperature 0, and that the same holds when you self-host with open-source servers such as vLLM or SGLang.
- That result is used to knock down the widely repeated hypothesis that GPU parallelism plus floating-point non-associativity is what makes inference vary, which the lab calls incomplete rather than wrong.
- The lab lays out four claims that are true at once, including that every kernel in a model's forward pass is deterministic while the person querying the server still sees nondeterministic output.
- Non-associativity itself is illustrated with base-10 arithmetic, where (0.1 + 1e20) - 1e20 gives 0 and 0.1 + (1e20 - 1e20) gives 0.1.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- decision If the deterministic-kernel claim holds, irreproducible eval numbers are a serving-stack problem to be filed against whoever owns the inference layer, not a property of the model to be averaged over.
- capability A four-line matmul loop gives any team a cheap way to check whether their own hardware clears the bar the folk explanation says it cannot, before they argue about kernels.
- constraint Bringing weights in-house does not buy reproducibility, so contracts and audit procedures that lean on self-hosting for repeatable answers are leaning on the wrong control.
- exposure The material offers no divergence rate and no throughput cost for a fix, so anyone budgeting for reproducible serving is currently pricing a mechanism rather than a measurement.
The diagnostic that carries the argument is four lines of PyTorch. Multiply two 2048x2048 bfloat16 matrices on a GPU, then do it 999 more times on the same data, and every result is bitwise equal to the first [5]. Concurrency and floating point are both fully in play here, yet the variation is not. So whatever explains a chat endpoint returning different text at temperature zero has to be something other than the mere coexistence of parallel hardware and finite precision [4].
Thinking Machines Lab's framing of the problem is the useful part, because it separates four statements that people usually collapse into one. Some GPU kernels are nondeterministic; the kernels in a language model's forward pass are not; an inference server's forward pass can be called deterministic; and the user of that server still sees nondeterministic results [6]. All four hold at once, which is why the folk explanation feels unfalsifiable. It attributes the user-visible symptom to a layer that, tested directly, does not produce it.
Non-associativity is still the reason numerical differences exist at all. The lab's own illustration: (0.1 + 1e20) - 1e20 evaluates to 0, while 0.1 + (1e20 - 1e20) evaluates to 0.1 [7]. Change the order in which a reduction accumulates and you change the last bits. That makes divergence possible, but it does not by itself cause divergence. Something has to change the order between two runs of the same prompt, and the same-input matmul test shows that the scheduler alone does not [5].
Sampling is the other explanation worth clearing out of the way. Ordinary variation in LLM output comes from sampling a probability distribution over tokens, which is genuinely stochastic [1]. Set temperature to zero and the model takes the highest-probability token every time, so the sampling step becomes deterministic in theory [2]. It does not become deterministic in practice, and that holds for hosted APIs and for self-hosted vLLM or SGLang alike [3]. Running your own weights on your own hardware does not buy reproducibility.
What is missing here is the size of the effect. The excerpt available here does not quantify how often two runs diverge, how far apart the resulting texts get, or what a determinism-preserving kernel costs in throughput. Batch-size dependence is the direction the lab's own title points, and its diagnosis rules out the alternative it names; the accounting for the fix is not in the material at hand. Treat "bit-identical inference is achievable" as a claim with a plausible mechanism behind it and an unpriced bill.
Where it already changes practice: if you are running an evaluation and your numbers move between runs at temperature zero, the honest read is that your serving stack is a variable in the experiment, not that the model is noisy. That is a testable distinction, and the four-line matmul loop is close to the cheapest test in the field.
What to watch
- A published throughput comparison between batch-invariant kernels and current defaults, which is what determines whether reproducibility is affordable in production serving.