Build1 distinct publisher3 min readPublished
Todd Linnertz borrows mutation testing for code review: plant defects, count the catches, and watch detection sag as AI-generated PRs climb. The arithmetic he offers resolves collapses at 40 canaries a window and nothing subtler.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The arithmetic behind the canary count is the standard binomial worst case, and it is worth redoing before you spend reviewer hours on it. Detection rate is a proportion, so the 95 percent half-width is 1.96 times the square root of p(1-p)/n, and it is widest at p = 0.5 [11]. At n = 30 that is 17.9 points. At n = 100 it is 9.8 [1]. Those are the 18 and the 10 Linnertz quotes, which means the figures in the post are the honest worst case rather than a flattering one [11].
The cadence he recommends is one canary per twenty pull requests in four-week windows, which for a team reviewing 200 PRs a week is 40 canaries a month [12]. At n = 40 the half-width is 15.5 points [2]. Comparing one window against the next adds the variances, so the worst-case interval on the difference is about 22 points [3]. A drop from 80 percent detection to 50 percent clears that with room to spare; a 5-point drift never will, which is why he calls the method a cliff detector and says gentle slopes stay invisible [13].
That makes the cadence a claim about your PR volume more than about your review process. Reaching 100 canaries inside a four-week window at one in twenty needs 500 PRs a week [5]. A team at 50 PRs a week gets ten per window [6], and ten, he writes, only confirms what you already believed [16]. So below a few hundred PRs a week you either seed heavier and pay in reviewer attention spent on defects you planted [17], or you widen the window and accept that your ceiling is an average over a quarter in which the codebase and the models both moved. (Two weeks of honest baseline numbers beats a quarter of arguing about which numbers to collect, he notes, on the grounds that the arguing is more fun [18].)
The number also depends entirely on canary quality. His examples are an off-by-one, a swallowed exception, and a permission widened exactly one notch, the kind of diff that reads fine at 4:40 on a Thursday [9]. Write them cleaner than that and high detection is measuring your canary author, not your reviewers. Two operational details the post leaves open decide whether the program is runnable at all: who writes the canaries, and what guarantees a seeded defect never merges [19]. Whatever that safety path is, it is also the leak, because the team is told the program exists but never which PRs are in it [10].
Storey's cognitive debt is the balance being accrued, the state where nobody can confidently explain how the system works or predict what a change will break [3]. Detection rate is the only meter in this proposal that moves in the same direction as that balance, since merge rate and time-to-approve both improve while it grows [5]. That is the argument for running it. Against it: Linnertz has not run it at a real team's volume and says so [14], and his evidence for the failure mode is his own project sitting on two months of green CI over conformance checks that had never once passed, caught by a human audit [7]. That shows instrumentation can lie, but it does not yet show that seeded canaries would catch it.
Ranked by verification strength, evidence, and original report placement.
Detection rate is a proportion, so error bars shrink with the square root of the count: thirty canaries in a window gets within 18 points at 95 percent confidence, a hundred narrows that to 10 points, and these are worst-case widths that tighten as detection climbs away from a coin flip.
Linnertz would start at one canary per twenty pull requests, measured in four-week windows; a team reviewing 200 PRs a week hits 40 canaries a month at that rate, which he says is enough to compare one window against the next.
At 40 canaries a window you can see a collapse, detection dropping from 80 percent to 50 percent, but you cannot see a 5-point drift; the method finds cliffs and gentle slopes stay invisible until they stop being gentle.
Linnertz writes that ten canaries tells you nothing you did not already believe.
Seeding heavier makes the signal arrive sooner, paid for in reviewer hours spent on defects the team planted itself; seeding lighter delays the first usable number.
The author, Todd Linnertz, sat in on a DORA community session presented by Margaret-Anne Storey.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · September 2, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Coverage at 80% was a price on human attention, and CI is the wrong place to charge it1 distinct publisher
build
Coverage Is A Line Counter, So A Coverage Gate Buys You Line-Counting Tests1 distinct publisher
build
The check that never fires: why every agent-built detector needs a negative control1 distinct publisher
build
Deleting guard lines one at a time found 40 of 61 unmeasured by any test1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Sound arithmetic, single uncorroborated voice
Two very different kinds of support sit side by side. The statistical core survives an independent recompute — 17.9 and 9.8 points at thirty and a hundred canaries, which is what Linnertz rounds to 18 and 10 — and the window-to-window comparison behaves as he says. Everything else is one man's testimony on his own blog: the two months of green CI, the thirteen-post retrospective, the read on Storey's session. No second account exists to check any of it, and the method itself has produced no measurement anywhere.
Nobody has run it, the author included
This is a proposal at the drawing-board stage, and the piece is candid about it: no team has published a detection curve, Linnertz has not run canaries at real volume, and his plan to fold the measurement into AIEOS is stated as intent rather than something already answering queries. The closest thing to uptake in the whole story is his request that whoever tries it first publish the numbers.
Self-limiting, but priced as an instrument
Linnertz does more to deflate his own idea than most authors would: he says the method finds cliffs and misses drift, calls the numbers arithmetic, and warns that anyone promising 5-point sensitivity is selling something. That discipline pulls this close to honest. What tips it slightly over is packaging — a headline that tells readers they have a ceiling, an instrument presented as ready to adopt while the queue it needs is 5 percent planted defects of unnamed authorship, and a minimum canary count that quietly excludes any team under a couple of hundred pull requests a week.
Building the product he says vendors can't sell
The disclosure is right there in the text — Linnertz is building this measurement into AIEOS — and it sits two sentences from his line that no vendor can sell a team its own ceiling. That is not concealment, but it is a builder making the case for the instrument he happens to be building, published on a platform with no editorial layer between the argument and the reader. His call for fifty teams to publish anonymized curves would also populate exactly the comparison set his own tooling would want.
Clear about what it says, thin on what it proves
We can read this one with reasonable assurance because the text is unambiguous, its numbers can be checked, and its author flags his own limits — an unusually easy piece to assess fairly. Confidence stops short of high for the obvious reason: one publisher, one voice, zero field data, and two operational gaps that would decide whether any of it survives contact with a real review queue.