Build1 distinct publisher3 min readPublished
Measured five times on the same plan, one safety critic returned five different verdicts and still never approved a seeded defect. It holds because the code downstream never reads the model's severity label.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The downgrade happens in one branch. If severity is BLOCKER and the finding's heuristic_family is not in a four-name frozenset, severity becomes WARNING before the finding enters the findings list [6]. The model's severity string is read and then overwritten before the finding is stored. The field doing the work is heuristic_family, and the author is explicit that the gate derived it deterministically, not from the model's output: "The LLM's label is decorative; the family is load-bearing" [7]. Port this pattern with family populated from a model field and you have moved the LLM one layer down rather than out of the path. The check still runs, but it is now checking a guess.
The two numbers either side of that fix belong on the same axis. Prompt engineering was tried first, and the critic still escalated completeness concerns to blocker about 30 percent of the time [8]. After the frozenset, zero advisory findings appeared as blockers across 92 post-fix runs [9]. Thirty percent of 92 is roughly 28 [2]. Those are not the same runs, so it is not a controlled before-and-after, but it is the order of magnitude the four-line branch removed.
Five trials deserve a caveat the post does not make. Five runs on one input give four consecutive comparisons, so label_flip_rate = 1.0 is a statement about four observed changes [1]. Four changes across four comparisons rule out any assumption of stability, but they say nothing about how far the verdicts drifted, whether the critic was maximally inconsistent or just inconsistent. The distinction does not need settling, because the architecture treats every degree of instability the same way regardless.
For underclaim_approvals = 0 to transfer to your system, your dangerous direction has to be structurally checkable. The gates named are preconditions, topological ordering, rollback credibility and verification ordering, and they parse the plan's AST instead of its prose [4]. The claim that they resist prompt injection follows from the same property rather than from any hardening: they never read natural language [5]. A seeded defect like a precondition with no earlier task establishing it has a right answer in the graph, so a critic that misses it on trial three loses nothing [10]. A defect like "this rollback will take four hours in practice" has no such answer. If your seeded defects are the second kind, the zero belongs to the author's test, not to your system.
I would take this trade in my own context, with the scope stated. The author's split puts code in charge of ordering, preconditions, rollback presence and schema, and leaves the model advisory on severity calibration, completeness and whether a rollback is credible [12]. The portable part is narrower than the principle: an enumerated set of finding families that code can adjudicate, and a rule that the model's severity string is never the field anything downstream reads. Severity from a model is a suggestion delivered in a confident voice.
Ranked by verification strength, evidence, and original report placement.
The author measured his LLM critic on identical input across five trials and it returned a different verdict every time, recorded as label_flip_rate = 1.0.
The author states the gates cannot be prompt-injected because they do not read natural language.
The author writes that the code trusts the structural property of which family a finding is in, which the gate derived deterministically, and that "The LLM's label is decorative; the family is load-bearing."
After the frozenset fix, zero advisory findings appeared as blockers across 92 post-fix runs.
The author argues that keeping the critical path deterministic gives resistance by design, but only to the things code can check structurally.
Across the same measurement, the critic never let a defective plan through: underclaim_approvals = 0.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 31, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
A gate that stops firing shows up in PlannerCritic's metrics as safer plans1 distinct publisher
build
The refusal came back labelled "infeasible": reading a planner/critic jailbreak test1 distinct publisher
build
A 170-goal agent field test costs $0.49. Proving it actually passed costs more.1 distinct publisher
build
One argument in a fix commit put the audited receipt in charge of its own verdict1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Code shown, numbers narrated, nothing reproducible
The mechanism is verifiable on its face — the frozenset and the downgrade branch are printed in full, and anyone can reason about what they do. The measurements are not. The post tells you to read the first two rows of a table that never appears, cites no repository or log, and supports its own history by pointing at earlier instalments of the same series. Strong on construction, weak on anything a second party could re-run.
One developer's own harness
Usage begins and ends inside PlannerCritic. What exists is real engineering activity — a versioned release, numbered issues, 92 recorded runs, a queued follow-up in v0.3.0 — but no other person, team or system is described as running any of it, and the only outside voice is a commenter who poked a hole in the argument.
Self-limiting, with one reach
The dek promise — five different verdicts, zero missed defects, safe because the code ignores the label — is fairly earned within the project, and the author spends his closing paragraphs naming two ways his own design leaks rather than hiding them. The overreach is the jump from that to a principle he thinks generalises beyond his project, made from a single codebase whose numbers nobody has seen. Small gap, and it sits in the generalisation, not the measurement.
Series promotion, but the numbers cut against the author
This is a companion post for the author's own series, on a platform where publishing is the point, and it cross-sells Articles 2 and 5 while pitching a design principle attached to his project. Against that: the two headline findings are a bug in his earlier prompt and total instability in his own critic, and he elevates a commenter's objection instead of burying it. No sponsor, employer or commercial interest is disclosed either way.
Internally coherent, externally unchecked
We are fairly confident about what the code does and what the author believes, and much less confident about the numbers attached to it. The mechanism is legible, the argument holds together, and the disclosed seams make the account more credible rather than less — but a single self-published writeup with an invisible results table and no replication caps how far this can be leaned on.