Build1 publisher3 min readPublished
Raw logs caught an LLM review debate replaying pre-generated text
A two-model code review produced rebuttals and a clean verdict while the raw logs showed no changed positions and no new evidence. The rebuild enforces independence by withholding each verdict until both are committed.
The Engineer · Build desk
What happened
- A developer pointed two LLMs at the same pull request and asked them to argue, and the transcript came back with confident claims and point-by-point rebuttals before a clean verdict.
- Reading the raw logs showed the second model replaying pre-generated text, with messages exchanged, no positions changing and no new evidence cited; the author says 89% of what was built was theater.
- The rebuilt system, AdversarialDebate, runs the two models in parallel with zero shared context, so neither sees the other's verdict before committing its own with structured claims and evidence.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Once Model A's verdict is in Model B's context, no instruction to be independent removes the anchoring, so teams running sequential review harnesses have to change the order in which context arrives, not the wording of the prompt.
- exposure A team that judges an agent pipeline by how well its transcript reads can ship a recording of a debate, because the failure shows up only in raw output.
- cost Adoption cost sits in ground-truth construction and harness debugging: the accuracy figures only became meaningful after 13 data-plumbing bugs were fixed, so the budget line is engineer time on labels, not API credits.
- contradiction Two of the author's own explanations for which model pairs argue well conflict, and both rest on one person's unreplicated runs, so model-selection advice from this work is a hypothesis to test in-house.
Anchoring here is a question of ordering. In the usual multi-model setup, Model A reviews the artifact, Model B reviews the artifact plus Model A's output, and the pair gets called independent review [4]. "The moment Model B's context contains Model A's verdict, B is anchored. It's doing social-pressure resistance, not analysis," the author wrote [5]. The fix in the rebuilt system is a commit barrier: Model B cannot see Model A's output until B has fully committed its own position [6].
A bounded debate opens after that, under two citation rules: every objection must reference a specific counter-claim, and every claim must cite specific text from the artifact [7]. Those rules give a log reader something to check. An objection either points at a counter-claim or it does not, and cited text either appears in the artifact or it does not [7]. The reported theater rate across 217 debates in v0.2.0 is 0% [9]. The post does not define how theater, or a binary match, is scored [22].
When the two reviewers cannot resolve, the output is a structured disagreement report, with both positions preserved and unresolved points documented [8]. The author argues that two independent reviewers reaching different conclusions from different evidence is itself the useful output, and that most multi-agent systems are engineered toward consensus [8].
The accuracy figures are claims about one corpus. In the first field test, 81% of debate claims matched documented PR outcomes across 411 debates on 70 pull requests from public repos [10]. For that to hold on your repository, your labels have to be as good, because the ground truth is what the pull request itself documented going wrong, and your bug classes have to resemble the ones in those 70 repos. By v0.2.0 the number was an 88.7% binary match across 150 artifacts in four domains, scored against a corrected 2,333-row dataset [12].
The correction was substantial. Before trusting any of the numbers, the author fixed 13 bugs, among them CSV parsers breaking on commas in PR descriptions, model slugs silently routing to the wrong provider, and a join that collapsed the dataset from 2,333 rows to 359 [11]. The broken join left about 15% of the rows [20]. None of the 13 were architecture bugs [11].
At v0.2.0 the run cost $0.42 for 360 reviewer runs [13]. That is roughly a tenth of a cent per run [21]. "Less than lunch. Compute was never the bottleneck. Prompt engineering and ground-truth measurement were," the author wrote [14].
Model pairing is the least settled part of the write-up. The best pair was GPT-4o-mini with Mistral Small 3.2, a small OpenAI model and a small European one [15]. GPT with Gemini produced the worst debates: many rounds, almost no concessions, no resolution [16]. The author's first explanation was diversity of training objective over raw capability, and a later run suggested the effect may instead be non-Mistral models rubber-stamping each other, which turned the working rule into "always include Mistral" [18]. "I have not proven this rigorously," the author wrote [19].
What to watch
- Whether a second operator reproduces the 0% theater rate on their own repositories, with the theater scoring script published.
- Whether the 88.7% binary match survives on private repositories where pull request outcome documentation is thinner than in the 70 public ones.
- Whether the "always include Mistral" rule holds when Mistral is swapped for another model trained outside the big US labs.