One Docker log fixture carries the entire margin in a thirty-run local-model benchmark

The gap is three runs. Four of the five fixtures agreed completely across both models, so they rank nothing, and the fifth came out 3 of 3 against 0 ^[21][22]. Three differing trials out of thirty puts 10 percent of the run budget under the whole result ^[24]. That supports one narrow sentence: on this machine, qwen3-coder-30b pulled every ERROR line out of a Docker log into structured JSON three times out of three, and qwen3:8b did not manage it once ^[22]. It does not support the claim under test, which holds that "a coding model treats rules as the whole point" ^[2]. The author's own verdict is that the result was clean and confirmed the article it was testing, but it was wrong ^[23].

The two sides are also not matched for size. The identifiers are qwen3-coder-30b and qwen3:8b ^[4], a 30b against an 8b ^[25], and log extraction into JSON is the kind of task where parameter count tends to show up. Training objective and model size are confounded, and one fixture cannot separate them.

Two sampling decisions set the conditions under which the number transfers at all. Thinking mode was off, because on this box the qwen models either bury the answer in a thinking block or get rejected by the server outright ^[11]. Temperature was left at the default instead of the low value used for code generation, deliberately, because the claim was about ordinary use rather than tuned precision ^[12]. If your pipeline edits structured text at low temperature, this run measured a configuration that is not yours.

The part worth copying sits upstream of every number. Gates are full equality or parse-and-compare, never substring, because a substring check asking whether the output contains the right value also passes output that contains the right value and three wrong ones ^[9]. Then the grader gets graded: a --selftest mode feeds known-correct answers to each checker with no model in the loop, before any request goes out ^[13]. It failed on YAML ^[14]. The fixture's frontmatter carries an unquoted date, 2026-07-01, and YAML's parser returns that as a datetime.date rather than a string, so the comparison against "2026-07-01" is false forever ^[15][16]. Six correct answers would have been marked wrong, two models by three trials, and the fix is a str() in the line guarding the untouched keys ^[17][26][18]. The writeup makes the general point that grading logic is code nobody reviews, because it lives in the part of the experiment everyone treats as infrastructure ^[19].

The chat template check ran on the same principle, and it is worth being exact about what it did. The template was read out of the runtime's own API before the general side was chosen, and came back as 1,723 characters of real ChatML ^[5]. Another model on the same box ships a template that is just a passthrough, which silently degrades everything sent through it, and the author's note is that picking that one by accident would have produced a fake result large enough to publish ^[6]. The template did not corrupt this comparison, because the failure was priced before the run and caught in advance. That is the kind of input a benchmark table never shows you.

Thirty runs bought one defensible ranking, on one task type, at default temperature, between two models of different sizes. The two pre-run checks bought considerably more.