Repeat runs rule out chance as the source of position bias in twelve LLM judges

Repetition stability is the control that makes the other two numbers readable. The measurement re-runs an identical prompt through the same judge, which separates ordinary sampling variance from an order effect ^[20]. In the paper's words, it "ensures that the observed position bias in the given model and tasks is not due to random variations" ^[7]. Without it, run-to-run flakiness and a genuine order effect land in the same column.

That control costs calls. Position consistency, which the paper calls one of the most widely used ways to measure the problem ^[9], cannot come from a single judgement. You need the verdict with a candidate placed first and the verdict with the same candidate placed second. That is two judge calls per pair ^[17]. Add one repetition in each order for stability and the pair costs four, or four times a single-order, single-run judgement ^[18].

Divide the reported total. Twelve judges over 22 tasks is 264 judge-task cells, so more than 100,000 instances spread evenly across them is roughly 380 per cell ^[16]. A per-task consistency rate therefore rests on a few hundred comparisons. The candidate answers came from about 40 solution-generating models on MTBench and DevBench ^[4], and the paper sorts its explanatory factors into judge, candidate and task levels ^[10]. For a consistency number here to predict yours, your task mix and your candidate pool have to look like theirs.

The strong effect sits in the quality gap between the two solutions, and the length of the prompt components is only weakly involved ^[6]. Direction is what an A/B harness needs, and the posted abstract and introduction give the strength of that relationship without its sign ^[19]. If smaller gaps produce more order-driven flips, then the near-tie between two good models is both the hardest case for a judge and the most common reason to run one. The candidate-level factor category is where that would show up ^[10].

Mitigation already has a record. The paper attributes four failure modes to prior work: incomplete bias elimination (Guo et al., 2024), increased complexity (Li et al., 2024b; Khan et al., 2024; Chua et al., 2024), the introduction of new biases (Ohi et al., 2024), and inconsistent effectiveness (Gallegos et al., 2024) ^[12]. The reason to keep judging with models is in the same introduction: human assessment is the gold standard for alignment with human preferences but lacks scalability and reproducibility at volume ^[14], while earlier studies report a high level of agreement between LLM judges and human judgments ^[13]. The authors also report that agreement and disagreement across the judges maps how judging difficulty is distributed over the dataset, and they say this points to potential dataset modifications ^[15].