Build1 publisher3 min readPublished
The orchestrator refuses to dispatch an implementation task until a research brief exists
An autonomous coding setup on a Mac mini splits reading from writing into two agents with separate tool lists. Its operator reports guessed-API failures falling from about one task in five to one in forty.
The Engineer · Build desk
What happened
- The implementer confidently called client.batchUpsert() on a library that never had it; the real method was upsertMany(), added in v4.2 with a different argument shape.
- The fix was a read-only research sub-agent whose tool set is Read, Grep, Glob, web search and web fetch, with no Edit, no Write and no Bash.
- Over three months the operator reports hallucinated-API bugs falling from roughly one task in five to about one in forty, at under 10 percent extra tokens.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Withholding Edit, Write and Bash puts the read-before-write rule in the harness instead of the prompt, so it survives the context pressure that ate the instruction after two days.
- cost The token overhead is charged on every task, while the saving only lands on the minority that would have guessed an API. A team with few such failures pays the tax and collects little back.
- decision Copying this means choosing the confidence number at which work is allowed to proceed, because anything under the bar sends the task back to research.
The gate is a file check. Before handing work to an implementation agent, the orchestrator looks for a research brief for that task with a confidence score in it, and will not dispatch without one [11]. Missing brief, or confidence too low, and the task goes back to the researcher instead of forward to code [12]. The prompt version of this idea had already been tried: "please read the docs first" helped for about two days, then the instruction got crowded out by everything else in the context window [7].
The other half of the design is tool denial. The researcher is defined as a Markdown file with frontmatter, and its tools line lists Read, Grep, Glob, WebSearch and WebFetch [18]. No Edit, no Write, no Bash [9]. The author's point was that it physically cannot dash in and fix something quickly itself [10]. The implementer already had web fetch and file tools of its own; it started producing code anyway [6].
The researcher's instructions put the lockfile ahead of the documentation. Check the pinned version first, because docs for the wrong major version are worse than no docs. Prefer installed source in node_modules or site-packages over web pages, and search the repo for an existing wrapper before answering [14]. In the sample brief, the pinned version was 4.3.1, the working call was `Model.bulkCreate(rows, { updateOnDuplicate: [...] })`, and `upsertMany` turned out to be a community plugin the installed version never had [15]. The verdict pointed the implementer at `insertBatch()` in `src/db/batch.ts`, which already chunks at 500 rows and has two callers. Confidence 0.9. Sources read: four files, one changelog, zero web pages [16].
Roughly one in five is 20 percent and about one in forty is 2.5 percent, so the reported change is 17.5 fewer hallucinated-API failures per hundred tasks [19]. In the case described, one of those failures cost three self-healing attempts and about 40,000 tokens before the task was marked blocked [4]. Treat 40,000 as typical and the avoided spend is 17.5 x 40,000, or 700,000 tokens per hundred tasks [20]. The briefs are reported at under 10 percent of tokens [17]. Across a hundred tasks that is ten times the average task budget, so the overhead clears at any average below about 70,000 tokens a task [21]. Both inputs are soft: 40,000 comes from one illustration, and the 10 percent figure is an upper bound given without a breakdown [4][17].
For the rate to transfer, several things have to hold in the target repo: dependencies pinned in a lockfile, dependency source on disk to read, helpers already in the tree worth finding, and failures that are genuinely guessed APIs [14][16]. The 60 percent "didn't look before writing" attribution came from a two-week audit of failed tasks [5]. The one-in-five to one-in-forty change was measured over three months [17]. The post reports both rates without saying how many tasks they cover [22]. Where the agent is calling a remote HTTP service, there is no installed source to prefer over web docs and the pinned-version rule has nothing to check. That is the weaker end of the same design.
What to watch
- Whether the same gate holds for agents calling remote HTTP services, where there is no installed source for the researcher to read instead of web docs.
- A published task count and per-task token baseline. Those numbers would let anyone test the under-10-percent overhead figure against the 40,000-token self-heal cost.
- How often tasks bounce back to research on low confidence, and what threshold the orchestrator uses to decide.