Build1 distinct publisher3 min readUpdated
A 157-goal field test of an LLM plan-and-critique loop found failures clustered in three structural families. Swapping in gpt-4o changed the writing, not the dependency graph.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The taxonomy does not close. Unverified dependencies at 57 blockers, unsafe sequencing at 46, and weak rollback at 18 sum to 121 [4][5][6][13], against 132 blockers reported across 63 strict goals [3]. Eleven sit outside the three named buckets [13], which is awkward next to the claim that every strict goal that failed did so because of one of the three families [19]. Either those eleven are extra instances inside goals already accounted for, or there is a fourth family that has not yet earned a name.
The proposed fix has a similar seam. A deterministic precondition closer, run on the draft to verify that every declared precondition is produced by some earlier task, is credited with removing 64 of the 132 blockers [10][11]. But unverified dependencies only number 57 [4], so seven of that 64 have to be drawn from the sequencing pile [14]. That is coherent rather than sloppy: once you know which task produces `replica_verified`, you also know the backfill cannot legally precede the quality gate that establishes it [5]. Dependency closure and ordering are one check read in two directions. Rollback gets nothing from the pass [6], and 68 blockers survive it [17].
The reason parameters do not help is visible in the rollback example. In `db-10-multi-tenant-split`, the dual-write task did have a rollback; it switched back to single-write and left the inconsistencies that dual-write had introduced [22]. That is not a model that lacks the word "rollback." It is a model that generates each step against local plausibility while the property being violated is global to the plan. Author Debashish Ghosal reports testing gpt-4o as planner with a mini critic and then for both roles, and got the same defect pattern with better prose [7]. The revision loop behaves the way you would expect from a local generator: it fixes one blocker and introduces another, reshuffles order without closing the gap, adds rollback to the wrong task [8].
What makes the diagnosis load-bearing is the critic's stability. The same blockers were found again across revisions, so the loop's failure is attributed to the planner's inability to structurally repair rather than to reviewer drift [9]. Without that, planner incapacity and critic noise would be indistinguishable, and the convergence detector firing at a median of two revisions across 33 strict goals [8] would read as flakiness instead of a wall.
Scope, honestly stated: this is one engine [1], one author's labels, one goal set. Strict goals were 63 of 157 [16], and the failing ones averaged 2.1 blockers each [15]. The three families are a finding about that harness until someone reproduces them elsewhere. The claim that survives regardless is narrower and more useful than the headline: a plan is a graph with checkable properties, and checking is cheaper than hoping the next checkpoint has learned to plan.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The author tested gpt-4o as planner with a mini critic, and gpt-4o for both roles, and reports the same defect pattern with better prose.
The critic reliably found the same blockers across revisions, so the failure is attributed to the planner's inability to structurally repair rather than to the critic's judgment.
PlannerCritic is an open-source engine in which one LLM writes a plan and a second LLM reviews it; the article is the third in a series about building it.
The field test described in article 1 of the series covered 157 goals.
The field test produced 132 concrete blockers across 63 strict goals.
Unverified dependencies accounted for 57 blockers: the plan declares a precondition that no earlier task establishes.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Specific but single-source and self-graded
The post is unusually concrete for a self-published engineering write-up: named goal corpora, per-family blocker counts, three verbatim critic outputs, and a stated median revision count. That specificity is what lifts it above anecdote. It is capped, though, by being one self-authored source with no external replication, no linked repository or blocker dataset, no prompts, and no ground-truth labelling of critic output — plus two unreconciled arithmetic gaps in its own numbers (121 of 132 blockers attributed; a 64-blocker fix aimed at a 57-blocker family).
Author's own project only
The only observable usage is the author running his own open-source engine over his own goal corpus, plus a model-substitution trial in the same harness. There are no third-party deployments, no user or install figures, no downstream integrations, and the proposed precondition closer is described as a plan rather than something shipped. Scored low on observed adoption, not on merit.
Mostly disciplined, two overreaches
The framing is more self-critical than promotional — the author documents his own system failing and explicitly rejects the bigger-model narrative — which keeps the gap small. It is positive rather than zero because two assertions run ahead of what is shown: the exhaustiveness claim that every failing strict goal fell into one of three families sits against 11 unattributed blockers, and the 48 percent improvement headline is an unimplemented projection presented in the same register as the measured counts.
Self-published series promoting own project
Observable incentives are moderate and structural rather than hidden: the author is the builder of PlannerCritic, the post is instalment three of a five-part series that cross-references articles one and two, and it is published on a self-publishing developer platform with no editorial gatekeeping. That rewards a memorable framing and continued reader attention. Offsetting factors: the content is a critical post-mortem of the author's own system, and no commercial product, pricing, or funding interest is disclosed in the supplied material.
Low-moderate: one self-reported account
Confidence is constrained by cluster shape more than by content quality. A single publisher and a single author-participant source, no artifacts to verify, no replication, and unresolved internal arithmetic mean the directional finding (structural planning defects persist across model tiers) is plausible and well-specified, while the individual figures should be treated as one team's self-reported measurements.
build
Dual 3090s, no NVLink: the serving stack broke long before the model did1 distinct publisher
build
Three manual interventions in a month, and every guard was working as designed1 distinct publisher
build
Six MariaDB versions, one real difference: the only reason to leave 10.6 is the July 2026 clock1 distinct publisher
build
Force the tool call, then hand Lightsail a long-lived key1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 22, 2026