Build1 distinct publisher3 min readUpdated
PlannerCritic's sweeps are cheap enough to gate every release. The harder problem is telling a clean run from a rig that quietly counted nothing.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Zero failures is a claim about the harness before it is a claim about the agent, and the v0.1.0 run of PlannerCritic [1] had four separate ways to return a number that meant nothing. Eighty-eight percent of the assertion files were in a format the runner did not recognise, and it returned 0/0 rather than an error [25][4]. The budget and replan dimensions also reported 0/0, because dispatch passed five arguments to a function that takes four [6]. The in-memory SQLite store reset between runs, losing cross-dimension state [8]. The results parser read `trace.get("status")` instead of `trace["result"]["status"]`, converting five passes into reported failures [7]. On the author's own breakdown, four of the ten findings were in the measuring apparatus rather than the planner [22]; exactly one was the sort of defect a conventional test suite would have called a failure [3].
That is the reason a green sweep needs something other than a pass rate to be readable. Three of those four bugs produced zeros or inversions that no summary line would flag. What distinguishes a clean pass from a silent one is the denominator: assertions evaluated per dimension, published next to the result, plus at least one goal that is supposed to fail and whose failure the rig has to report. Neither release note describes such a canary [14][15].
The economics are more interesting once you divide. v0.1.0 spent about $0.0019 per goal [19]; v0.2.1 spends about $0.0029 [20]. Per-goal cost rose roughly half while the suite grew about eight percent [21], and the source does not say why, though v0.2.1 added a live-critic boundary evaluator and an operational benchmark [15]. More usefully: the first run cost about three cents per issue found [24], and that metric goes undefined the moment the gate starts returning zero. What replaces it in the source is the assertion that a sweep is cheaper than debugging one production incident caused by a bad plan [27], which is not a number you can check.
Two findings put a floor under the price and a ceiling over the fix. Local Qwen3.5-4B and 9B models could not produce structured JSON [11], so the run cannot be moved off a paid API to get to zero marginal cost. And GPT-4o reproduced the same defect patterns as gpt-4o-mini [13], so the planner gap is prompt and schema work, not a billing decision. Meanwhile the critic is non-deterministic: strict goals re-run at `revision_cap=4` all escalated, with different blockers on each revision [12].
The accounting deserves one more look. v0.2.0 is described as 170 goals across 40 domains against v0.1.0's 157 across 35 [14][2], with 14 new goals in five new enterprise domains and three new adversarial-policy goals [17]. Seventeen additions on a base of 157 gives 174, not 170, leaving four goals unaccounted for [23]. The whole 10-to-0 arc [18] assumes the two suites are comparable, and a four-goal discrepancy in the published totals is the same class of problem as the assertion files: the number is stated, and nothing in the write-up demonstrates it was counted.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
57 of 65 assertion files were in the wrong format and the harness did not notice, quietly returning 0/0 results with no crash and no error.
In v0.1.0, a result of 0 failures meant the harness was silently broken; the author writes that a field test returning 0 failures should first be distrusted.
v0.2.0 ran 170 goals across 40 domains, fixed 31 bugs found in code review, and the field test itself found zero new issues.
The author states that code review before the field test cost $0 and 2 hours and found 31 bugs, meaning the field test validated fixes rather than discovering them.
The author states that code review caught all 41 bugs before the LLM ran, turning the field test from a diagnostic sweep into an immutable regression gate.
PlannerCritic is an open-source engine in which one LLM writes a plan and a second LLM reviews it.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed but wholly self-reported, single source
The account is unusually specific — named defects, file-level bug descriptions, issue numbers, goal and domain counts, and dollar figures — and its internal arithmetic mostly checks out (41 = 31 + 10; $0.49/170 goals). But every number comes from one dev.to post written by the project's own author, with no independent replication, no third-party audit, no raw run artifacts in the supplied material, and one unreconciled discrepancy in the goal-addition accounting. Evidence quality is therefore constrained by provenance rather than by detail.
Author's own release cadence only
The only adoption signals are the author's own shipping history — v0.1.0, v0.2.0 and v0.2.1 with expanding goal sets, a SWE-bench-derived security oracle and a deterministic test gate. There are no external users, downloads, stars, forks, deployments, contributors, or production usages reported anywhere in the supplied source, so adoption beyond the maintainer cannot be observed.
Modestly overstated framing over sound engineering detail
The underlying engineering content is grounded and self-critical, but the packaging overstates it: the headline pairs the v0.2.1 cost ($0.49 for 170 goals) with the v0.1.0 finding count (10 issues from 157 goals); 'zero field-test issues' across two releases is graded by the same author whose earlier harness silently returned 0/0; '$0, 2 hours' for code review excludes engineering time; the 'immutable regression gate' language outruns the single-maintainer, self-verified evidence; and the per-goal cost rose ~51% against ~8% scope growth without explanation. The gap is moderate rather than severe because the post explicitly warns readers to distrust 0-failure results.
Author documenting and promoting his own project
The post is article 4 in a series by the maintainer of the open-source engine it evaluates, published on a developer-audience platform. The author sets the test scope, writes the harness, grades the results, and reports the outcomes that make his release process look effective. That is a clear self-promotional and self-assessment incentive. Mitigating factors: no pricing, paid product, sponsorship, or funding is disclosed in the source, and the post volunteers substantial unflattering detail about its own harness bugs.
Single publisher, single self-interested source
One source, one publisher, one author who is also the subject. Claims are internally consistent and highly specific, which supports confidence in what was reported; but with no independent corroboration, no external adoption signal, an unexplained goal-count discrepancy and self-graded pass results, confidence in the broader conclusions about release-gate reliability stays low.
build
132 blockers, three defect families: the bigger model wrote better prose and the same bad plans1 distinct publisher
build
OpenClaw makes the channel the architecture, and the reasoning loop a lodger1 distinct publisher
build
The streaming edit is a software layer: proxies, stubs and an ffmpeg hook on storage you own1 distinct publisher
build
752 Nigerian institutions, shipped as a repo instead of an endpoint1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 24, 2026