Build1 distinct publisher3 min readUpdated
Eleven real vulnerabilities in roughly 300 pull requests over four months. The noise was bad enough that the fix was a second agent whose only job is refuting the first.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Eleven findings in four months works out to one real vulnerability per 27 pull requests [1], fewer than three a month [2]. That is the yield the whole apparatus has to justify, and it means the reviewer's normal output is silence. Anything that adds volume to that baseline buries it.
The bug class driving the build makes this worse. Broken access control sits at the top of the OWASP Top 10 and is invisible to pattern matchers, because the vulnerable code and the safe code are identical except for a missing ownership clause in the query [10]. Semgrep and CodeQL report known patterns well and return nothing at all for an endpoint that confirms you are logged in and never confirms the record is yours [7]. So there is no second opinion in the stack to grade the agent's findings against. The verifier agent is the only calibration available.
The mechanism worth copying is the incentive, not the model. An agent told to review a diff for security issues produced a wall of "consider validating user input" attached to code that already validated user input [17], because producing findings was the task. Two changes inverted that. The prompt became roughly 60 lines of enumerated threat classes, each requiring either a finding or an explicit "N/A" with a one-line reason per changed file, with silent skipping forbidden [18]. Then the second agent arrived, and its only success condition is refuting what the first one said [3]. Refuted findings are dropped and logged; survivors post as a blocking comment for a human to fix or dismiss [12].
Which leaves the number nobody published. The refuted findings are on disk, by the author's own description of the flow [12], so the raised-to-survived ratio exists. The post does not give it [3]. Without it, "caught 11" is a numerator, and the false-positive tax is the line an operator actually has to budget: a second model pass for every finding that dies, plus a human read for every one that lives [12]. That cost lands per push, which is why docs-only pushes skip the review entirely to avoid burning tokens [14], and why the gate is a plain pre-push hook plus a CI step rather than anything more ambitious [13].
The structural decisions are cheaper to adopt than the prompt. The reviewer is a separate agent with its own system prompt rather than another bullet in the implementation agent's instructions, and the author says that distinction mattered more than anything else in the build [11]. Its tool allowlist is read-only: read, grep, glob, and git diff, with no write access, no arbitrary shell, no push [15]. His reasoning is blunt: a reviewer that can edit code is not a reviewer, it is a second author [16].
The summary offered is that a language model can cover the gap pattern matching cannot, but only with an adversarial second pass [20]. Priced as a build, that means the unit of agent code review is two agents, and any plan that budgeted for one is wrong by an amount this data cannot tell you.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The reviewer 'cried wolf a lot' until the author added a second agent whose only job was to disprove the first one's findings.
The author's summary is that a language model can cover the gap pattern matching cannot, but only with an adversarial second pass.
The working prompt is about 60 lines and requires that for every changed file the agent check each enumerated threat class explicitly, stating either a finding or 'N/A' with a one-line reason, with no silent skipping of a class.
The enumerated threat classes include AUTHZ (ownership as well as authentication checks), INJECTION, SECRETS, SSRF/PATH, CRYPTO, DESERIALIZATION, and a rate-limiting class.
The author runs a fully autonomous implementation system in which agents pick up work, write code, and open pull requests largely without him.
The stated fear was not bad code, which tests catch, but code that works perfectly and is also a security hole: a happy-path integration test does not care that a user-controlled string was interpolated into a shell command, so it passes and ships.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed mechanism, self-reported results
The implementation is well evidenced for a single source: the pre-push hook, tool allowlist, pipeline diagram, threat-class prompt, and verifier prompt are reproduced verbatim and are self-attesting. The outcome side is weak — 11 vulnerabilities across ~300 PRs is one person's recollection with no logs, no examples, no baseline comparison against the Semgrep/CodeQL tools it is said to beat, and no independent corroboration anywhere in the cluster. The supplied body also truncates before the promised scorecard.
One developer, one repository
Observed adoption is a single practitioner's four-month run inside his own autonomous-agent codebase, deployed via a personal pre-push hook and CI step. No team, organisation, product, or second user appears in the supplied material, and no download, star, or usage metric is disclosed.
Mildly overstated but self-critical
The article is unusually candid — it names its noise problem, admits the naive prompt failed, and presents the verifier as a necessary correction rather than a triumph, which suppresses hype. The gap that remains is structural: a quantified win (11 real vulnerabilities) is paired with an unquantified cost (findings raised, findings refuted, tokens spent, misses), so 'it works' rests on a numerator without a denominator. The headline conclusion generalises a one-developer, one-repo anecdote into a pattern claim.
Practitioner credibility, no disclosed vendor tie
This is a self-published developer-platform post by an author whose subject is his own autonomous-agent workflow, so there is a reputational incentive for the experiment to read as a success and no editorial layer to challenge the numbers. Offsetting factors: no product is sold, no sponsorship or vendor relationship is disclosed in the supplied text, and the piece volunteers its own failures. Nothing in the sources establishes a commercial arrangement, so this is scored on visible publication context only.
Confident on method, weak on outcomes
High confidence that the described system exists and works as diagrammed, because the code and prompts are shown. Low confidence in the magnitude and transferability of the result: one publisher, one practitioner, one repository, self-reported counts, no precision data, and a truncated body that omits the scorecard the article promises.
build
A cleanup commit deleted the sanitizer. Five days later a scanner cashed it in.1 distinct publisher
build
Your reviewing model is reading the diff when it should be reading the session1 distinct publisher
build
Coverage at 80% was a price on human attention, and CI is the wrong place to charge it1 distinct publisher
build
The stability step is a branch, not a pipeline: inside one team's release-candidate discipline1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 22, 2026