Loading today’s stories
Build1 distinct publisher3 min readPublished
One developer counted the automated checks in his own repositories and found that only about one in nine could prove it was capable of rejecting anything.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Multiply the percentage out and the audit reads worse than the headline figure does. Roughly 182 of those 204 checks were drawing conclusions that nobody had ever watched them decline to draw, leaving about 22 with any demonstrated capacity to refuse [1][2][1]. That is one person's self-reported count in his own repositories, so treat it as a shape rather than an industry base rate.
The interesting part is which of the three gates does the work. A check can pass "broken repo goes red" and "fixed repo goes green" while keying on something incidental to the actual defect [3]. Gate three, replanting the original mistake and demanding red again, is the one the author says earns its keep: if the check still passes, it never understood the bug [4]. His evidence for that is uncomfortable and specific. Gate three has caught real breakage twice in guards written that same week for exactly this class of problem [5]. Authorial intent, in other words, buys nothing.
The same hole shows up in scoring rather than gating. The post cites @shreyasht abandoning a token-optimization project after his best number, 97% savings, turned out to come from a run that did no work at all: the agent asked a clarifying question, stopped, and the dashboard crowned it [6]. The correction was a denominator change, per solved task instead of per task [7]. A metric with no path to a negative verdict and a judge with no demonstrated veto are the same defect wearing different clothes.
Case invention is where this normally stalls, and the answer offered is cheap: feed recorded incidents that already carry a written "what went wrong" field to a model and have it emit runnable checks. Ten lessons produced ten cases and thirty gates, all thirty passing, none discarded [8]. A zero-discard rate on a sample of ten is not yet a yield figure. One failure in that run is more instructive than the successes: the first attempt tripped an internals filter because the example handed to the model had a product name in a comment and the model copied it faithfully [9]. The paranoid guard was written for sloppy humans and caught an obedient machine instead.
None of this survives the calendar, which the author concedes. Wiring a known-bad case proves the reviewer could refuse on the day it was wired, and says nothing about today [14]. Hence the proposal worth stealing: publish one date, the last time this reviewer refused something, next to the uptime number, run the known-bad case on a schedule, and let the date going stale be the alarm [13]. The reasoning comes from @james_anderson_h, quoted in the post: a checker nobody has ever seen fail is indistinguishable from one that approves everything, and the two produce identical logs until the rubber stamp waves through the thing that hurts you [11]. His sharper version is that silence and health look the same unless you deliberately build a state for "not verified lately" [17].
The author's own opening admission is the cost of not having that state. He filed a data-loss report about roughly 1,000 files that were never lost, and nobody, himself included, had a number to check it against [15].
Ranked by verification strength, evidence, and original report placement.
The author counted 204 guards across his repositories and found 89% had never been shown they can fail; of the automated checks that draw a conclusion, only about 11%, one in nine, could prove it was capable of failing.
Gate three has caught real breakage, including twice in guards written that same week to catch exactly this class of problem.
@wrobeltomasz independently described the same discipline in a comment thread: identify the checks, then run them "in simulation mode to confirm that they can indeed respond to invalid input", moving verification "from statistics in the README to actual system resilience."
The author proposes a "veto heartbeat": surface one date, the last time this reviewer refused something, first-class and visible next to the uptime number, with the known-bad case running on a schedule so that a last-refusal date aging past that schedule is itself the alarm.
The author's fix: every reviewer, whether a hand-written check, an LLM judge, or a second agent grading the first, gets one known-bad case wired through the live path, the same entry point real work uses, not a unit test beside the pipeline.
The benchmark harness runs three gates per case and a case that misses any of them does not run at all: unsolved state must go RED, solved state must go GREEN, and a deliberately replanted known-bad must go RED again.
Follow any of these and your For You feed starts watching them — no settings page required.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single first-party account, quantified but unauditable
All findings come from one dev.to post by the practitioner himself. The numbers are specific (204 guards, 89% unproven, three gates per case, ten incident-derived cases, thirty passing gates) but no repository, harness source, run log, or third-party audit is provided, and the two most market-shaped claims are relayed second-hand from commenters. The post's internal candour -- conceding the verification is point-in-time and narrating a false data-loss report against itself -- raises credibility without adding verifiability.
One practitioner's own toolchain, plus one peer describing similar practice
Observed adoption is confined to the author's own repositories and benchmark harness -- the three-gate discipline, the batch of guards made to prove failure, and the incident-history-to-cases pipeline -- with a single commenter independently describing simulation-mode verification. No team, vendor, product, or open-source project is reported to have shipped this pattern, and the veto heartbeat is a proposal with no deployment.
Generalized framing runs ahead of an n=1 evidence base
The framing extends a personal audit into an industry-wide diagnosis -- unexercised checkers equal rubber stamps, most tools ship audit logs instead of veto proof -- and asserts the remedy 'costs an afternoon and it works' without effort data or false-positive accounting, while adoption remains one developer's repos. The gap is moderate rather than severe because the post repeatedly discounts itself: it concedes the verification says nothing about today, reports that its own fix 'wasn't enough', and documents a self-inflicted false alarm.
Personal-brand publishing incentive, no disclosed commercial stake
The piece is a self-published developer post that continues a prior instalment on the same 204-guard count and coins a named concept, the veto heartbeat, giving the author audience and reputational incentives to keep the numbers striking. Against that, no product, employer, vendor, or funding relationship is disclosed or evident, nothing is being sold, and the narrative volunteers embarrassing failures of the author's own judgement, which cuts against pure promotion.
Coherent and self-critical, but wholly uncorroborated
Confidence is limited by structure: one publisher, one author, zero independent verification, and three key supporting anecdotes relayed without primary sources. The mechanical claims are internally consistent, specific, and plausible for the described setting, and the author's explicit scoping of what his evidence proves supports moderate trust in the practice descriptions while leaving the generalized diagnosis and the afternoon-cost estimate weakly grounded.
build
A guard that only speaks in exit codes cannot tell you it stopped guarding1 distinct publisher
build
Code review's quality gate survives AI. Its apprenticeship does not.1 distinct publisher
build
A £40 refund and a £40,000 one look identical to a pre-execution guardrail1 distinct publisher
build
Test count told you nothing about the merge gate; 17 attempts to beat it told you something1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 26, 2026