A noisy judge drags an adaptive agent stop rule below a fixed six-step budget

CDV separates the agent from its grader. Each step the agent claims is scored by a judge, and a policy reads the score history to decide whether to go again ^[10]. The note's author gives the reason: when an agent grades itself, the entity deciding to stop is the entity being judged, and agents optimise reported progress ^[22].

The policy is an ordered stack of guards, and the first guard to call a stop wins ^[11]. A score threshold sits at the top, followed by a plateau check that fires when the last three scores are within 0.01 of each other ^[11]. Next is a Bayesian guard that stops when learned expected improvement over the remaining steps cannot close the gap to the bar ^[11]. Budget, wall-clock, token and repeated-output limits come last ^[11]. The plateau and Bayesian guards let the adaptive policy stop before the threshold rule would ^[12]. With the threshold guard first, it can never stop later. In the June table the adaptive row and the threshold row are identical, and the headline compared the adaptive row with the fixed budget ^[13]. The plain rule was already there, the author wrote, "with nothing learned and nothing Bayesian about it" ^[23].

The simulation is small enough to reason about by hand. Each synthetic task has a hidden quality curve with diminishing returns, q(t) = 1 - (1 - s0)e^(-r(t-1)), with s0 and r drawn from easy, medium and hard task types ^[14]. The bar is 0.80 and the cap is eight steps ^[14]. The loop sees that curve plus Gaussian noise on each step, and the original run set it at sigma 0.02 ^[15]. The replication sweeps sigma from 0.02 to 0.20, over 10 seeds of 300 test tasks each ^[16]. The 0.10 setting is five times the original noise ^[4].

A rule that stops at the first passing score stops on whichever step the noise ran high. At sigma 0.10, a task one full standard deviation under the bar can print a pass on a single lucky draw ^[6][14]. The sweep catches this by recording true reach, whether hidden quality was at least 0.80 at the stopping step, next to observed reach ^[8]. The June benchmark measured only observed reach ^[8]. At sigma 0.10 the two differ by 28.5 points ^[1]. The fixed budget's 7% miss comes from the tasks alone ^[7][3].

Two-in-a-row confirmation lands 4 points above the fixed budget at that noise level ^[5]. It works in this simulation because each step gets its own noise draw, so a lucky score has to repeat before the loop stops ^[15]. For the result to carry over, a real judge's errors have to be roughly independent from one step to the next. A judge that keeps favouring the same wrong diff would pass the check on one mistake counted twice. The study calls no LLM anywhere; it tests the decision policy given a score ^[18].

The replication is careful work. The benchmark script had been removed in a repository cleanup, and the author restored it from public history ^[19]. Run against cdv at commit 356d496 on Python 3.12.3, it reproduced the June table exactly ^[3]. The note also tested a smooth-2 rule, a 0.85 margin threshold and a cold start with 0 to 300 warm-up tasks ^[21]. The excerpt ends before those results.

In my view, for a coding-agent loop graded by an LLM judge with unmeasured noise, two-in-a-row confirmation is a better default than the Bayesian stack. It needs no learning ^[21], and its failure case can be tested directly by measuring how often the judge repeats an error. The author wrote: "The published headline was true and beside the point: the decision is only as good as the score it trusts." ^[20]