Twelve divergences in 500 calls is a 2.4 percent rate [1]. Inverted, that is one divergence every 42 identical calls [2]. A test that makes one such call per run and diffs the whole output string will go red at roughly that cadence, which is the difference between a test that looks broken and a test that looks unlucky.
That rate is one measurement, on one endpoint, with one prompt, reported in a dev.to writeup [2]. Three conditions have to hold before it transfers to your suite. Your prompt has to land on a position where the top-2 logit gap is narrower than last-bit noise [19]. Your provider has to be batching you against co-tenant load that varies through the day [11]. And your assertion has to be sensitive to the byte [1]. Most positions fail the first test: after "the capital of France is", the gap to the runner-up is far too wide for a 1e-7 perturbation to close [18]. A pipeline whose outputs are mostly locked continuations can loop 500 times and see nothing.
The movement is upstream of the sampler. argmax over identical floats returns the same index every run [9]. But floating point addition is not associative, and the reductions in a forward pass (hidden-dimension sums, softmax denominators, RMSNorm) are split across GPU blocks and recombined in an order set by the tensor shape the kernel was handed [5][10]. Shape follows batch. The same request beside 48 others at 3pm and beside 3 at 4am gets a different tiling, a different split-K, and different last-bit rounding on its logits [11]. Greedy decoding stays deterministic throughout; what changes is the logits it is deterministic over, since those are no longer identical run to run [4]. Each run is fully deterministic given its batch, but the batch itself is set by the server and hidden from the caller [20]. The missing property has a name: batch invariance, a property most production inference kernels lack [6].
Batch size is not the only lever. Mixture-of-experts routing under capacity limits ties your token's expert assignment to whichever other tokens competed for that expert in the same batch [13]. Prefix caching moves the compute boundary depending on whether your prompt hit a cached KV prefix, which regroups the reductions after it [14]. Speculative decoding is built to be output-equivalent, but verification runs a different numeric path and the equivalence holds only to those same last bits [15]. Two GPU generations behind one endpoint means two kernel choices for identical input [16]. Fixing batch invariance alone would leave the cache and the verifier in the numeric path.
The remedy in the post is to assert on parsed fields and invariants, pin model snapshots, and keep a flake budget, on the stated ground that bitwise reproducibility exists only on your own hardware at batch size 1 [8]. Be clear about what that buys. Parsed-field assertions stop you failing on whitespace and reworded prose. They do not stop a refund ticket coming back billing instead of fraud [3], because the label is the parsed field, and the near-tie between fraud and billing is exactly where the classification lives [19]. Autoregressive decoding then carries the swap forward, since one changed token is the input to every token after it [7].
The flake belongs in the budget. The label flip does not, and it needs handling outside the model: if a single greedy call decides a refund, the tie-break should sit apart from the model, as a threshold on the parsed confidence, a second pass, or an explicit escalation path for the ambiguous class.
