157 plans, one real model: the expensive agent failures land before the first tool call

A developer writing on dev.to published a field test of 157 agent planning traces run against a real LLM and concluded that the failures worth caring about happened before the agent's first tool call ^{[1] [2]}. If that holds, the standard investment pattern in agent stacks is hardening a layer that was not the one breaking ^[3].

The mechanism he describes is mundane. An agent takes a goal like "migrate this service to the new auth provider," decomposes it in a single hidden chain-of-thought pass, and starts moving; three steps later the database schema was never checked, the outage window was never coordinated, or the rollback path was never real ^[4]. The cost is not a wrong answer. It is mutated state: by then you are not debugging the agent, you are cleaning up what it already changed ^[5].

The sharpest artifact in the piece is one trace, a blockchain recovery goal labelled bch-02-chain-split-recovery. The planner's first draft had four tasks: pause_attestation, identify_canonical, resync_node, verify_attestation ^[6]. The critic returned four BLOCKER findings, all of type unsafe_sequencing, one per task: pause ordered before its prerequisite detect_split, identify_canonical before pause, resync before identify_canonical, verify before resync ^[7]. Every step sat in front of the thing it depended on ^[8], which means every task in the plan was flagged ^[9]. The author's reading is that the planner knew the right steps but could not reliably order them, and that this is more dangerous than a dumb plan because the plausible one survives a glance at the task list ^[10]. The planner revised, the critic found the same blockers, and after two revisions the loop escalated ^[11].

The tool built around this, PlannerCritic, treats a plan like a pull request ^[12]: one model drafts, a separate model reviews, deterministic gates check structure, and the planner revises until the plan is approved or escalated to a human ^[13]. Two design choices travel even if you never install it. The deterministic gates run first and check ordering, branch sanity, rollback coverage, verification, preconditions and high-risk completeness, and because they never read the goal text the author calls them injection-immune ^[14]. The critic is a different model on purpose, because same-model self-review is too easy to fool ^[15]; his formulation is that a model reviewing its own plan is "agreement with extra steps" ^[16]. The loop is bounded by a revision cap, convergence detection and budget enforcement ^[17], and when it cannot converge it emits one minimal human question instead of guessing ^[18].

Now the evidence discount. The test plan defined 156 scenarios and produced 157 traces because one goal was renamed during the build, with all planned scenarios covered ^[19] - so the headline number is 156 planned scenarios plus one bookkeeping artifact, not extra coverage ^[20]. More importantly, the material available here stops before the aggregate results: we get the architecture and one trace, not a distribution of how many of the 157 drafts drew blockers, nor a false positive rate for the critic ^[21]. The author also cites existing research that self-correction fails often when a model cannot independently verify its own answer, as prior support rather than as his own measurement ^[22].

What to watch: whether the published field test results break down blockers by scenario and finding type, how often the deterministic gates catch ordering errors without a second model in the loop, and the escalation rate, because a planner that escalates constantly is a human review queue with added latency.