Build1 publisher3 min readPublished
Censoring infra failures shrinks the denominator behind a coding-agent pass rate
A dev.to post gives every failed coding-agent run one of four labels and scores skill only over the runs where the harness stayed healthy. SSH drops and disk-full errors get published as their own rates.
The Engineer · Build desk
What happened
- A dev.to post proposes giving every unsuccessful or unfinished coding-agent run exactly one of four labels, MODEL, HARNESS, INFRA or FLAKY_TEST, before any rate is aggregated.
- Its skill score divides MODEL successes by MODEL successes plus MODEL failures, censoring the INFRA, HARNESS and FLAKY_TEST rows and reporting them as their own numbers.
- Sixteen tasks is enough to debug the method and not enough to crown a winner, and that caveat is to be printed in the same paragraph as any rate.
- A job preamble records the host fingerprint, harness commit and suite hash before the first prompt, and a batch that starts with tight free disk is labeled INFRA and never run.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Every censored run leaves the denominator, so the published rate rests on fewer rows than were scheduled and a suite has to grow before a gap between two agents survives the split.
- decision Someone has to assign the arguable rows before aggregation, and the four definitions do not settle where a stale .pyc left by a dead job belongs.
- capability The empty-patch probe and the already-green controls test the grader in the same batch as the model, so a grader awarding unearned points shows up before it raises a score.
- cost The price of the honest number is per-run review time, because the classifier is a proposal to tune locally and the borderline rows still get read by a person.
Start with the denominator. Treat every non-zero exit as "the agent failed" and a broken pipe during git clone scores exactly like a wrong patch [10]. Label that run INFRA and it leaves the numerator and the denominator together, so the same logs yield a higher rate over fewer rows [6].
The post's opening scenario shows the size of the effect. Agent A finishes at 62 percent, Agent B at 81 [7]. Then syslog loads: four of A's misses are broken pipe during git clone, one is disk-full on /tmp, and two are pytest importing a stale .pyc after a previous job died mid-write [8]. Seven of the misses belong to the box [2]. The post does not say how many tasks the scenario ran, so take the 16 it calls enough to debug the method and not enough to crown a winner [14]. One task on a 16-task suite moves the rate 6.25 points [3]. The gap between A and B is 19 points, or about three tasks [1][4].
The bucket definitions still need a person to apply them. INFRA is defined as SSH drops, disk full, host OOM, provider 5xx, rate limits and wall-clock preemption [4]. HARNESS is a wrong runner, prompt template, grader or sandbox config, where the agent never had a fair shot [3]. FLAKY_TEST is settled by a rerun rule: the same task commit fails or passes with the agent disabled [5]. The stale .pyc row satisfies that rerun rule even though a dead job on the host caused it, and the post tells you to read the borderline rows yourself, since the pattern-matching script it ships is a proposal to tune on your own harness [23].
The grader-side checks are the part I would copy first. An empty patch against a red test must fail, and if that probe passes, the grader is broken [22]. At least four tasks in the holdout are negative controls that are already green, so an agent that "fixes" them by rewriting working code has found a grader bug [12]. The bytes_changed metric comes from git diff --stat, which stops a silent no-op counting as a win [15].
On the published figure, the rules are narrow: zero retries in the number itself, retries confined to an appendix, and decoding settings recorded as the API returned them, whatever the config said [18]. The post also sets a stop condition. "If infra_censor_rate is high, you do not have a model comparison. You have a hosting diary," it says, and the instruction is to fix the box or throw the batch out [16].
The post concedes the ground it can: the claim that models already outcode most developers "might hold on a clean bench", but not for a CSV that scores every non-zero exit against the agent [10]. Its verdict on shipping that column unlabeled is blunter. "Publish that column and you are doing marketing in a lab coat," the post says [9].
What to watch
- Whether the full labeling script ships, since what the post publishes is a truncated proposal.
- Whether any public coding-agent leaderboard starts reporting eligible_n and infra_censor_rate next to its resolved rate.
- Whether vendor bake-off write-ups adopt the zero-retry published number with retries confined to an appendix.