Skip to content

Build1 publisher2 min readPublished

A PM spent thirty minutes debugging a table ChatGPT invented for him

A dev.to post's case for harness engineering rests on one PM's half hour and one investigation where directed context cut a task from about 50k tokens to 15k, a figure the author himself calls an observation.

The Engineer · Build desk

Illustration accompanying A PM spent thirty minutes debugging a table ChatGPT invented for him

What happened

  • A product manager pasted a production query into ChatGPT and got back a table that does not exist in the company's database, then spent half an hour debugging the invented hypothesis.
  • The dev.to post defines harness engineering as the environment controlling how models use tools, receive context and access resources, and counts Cursor, Kiro, Codex and Claude Code as harnesses already.
  • In the mature version, sensitive actions stop for confirmation because the harness forces the stop, and every tool call is logged with cost metrics so the run can be reconstructed.
  • The team keeps a catalogue of around 80 steerings across four frontend repositories, with only a subset loaded into context for any given project or task.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The narrowing that saves tokens is not in the protocol. Whoever writes the tool schemas and response limits owns the token bill, so adopting MCP without designing the server buys the plumbing and none of the saving.
  • decision Letting a PM ask questions of production data turns into an access-control design: authorized views, a query allowlist, PII masking and row limits. That puts the schema owner in the loop before anyone writes a prompt.
  • cost Consistency across four repositories is now paid for as a catalogue of around 80 markdown steerings that someone has to keep current, plus the per-task choice of which ones load.
  • exposure Anyone putting the 70 percent token reduction into a business case is quoting a single unreplicated run that its own author calls an observation.

A model with no connection to the database cannot check a table name against anything, so it returns one that fits the shape of the query. The author wrote that the fault is not the model's; it sits in the missing harness [3].

For that case he prescribes a server. A product harness exposes authorized views through an MCP server, with an allowlist of queries, PII masking and limits on time and rows, so a PM gets business tools instead of arbitrary SQL [15]. He notes the PM did not know a database MCP existed [16]. Standing that up is an access-control review with a protocol in front of it, and it needs whoever owns the schema.

The token figure is one run. Directed context took an investigation from about 50k tokens to about 15k with the model and the task held constant [11], which is 35k fewer tokens, or roughly 70 percent [20]. He says he has not treated it as a benchmark, calling it an observation rather than a systematic measurement [12]. Each steering loaded is one question the agent does not ask again, he writes, because an agent that already knows the backend is PHP 8 stops asking [21]. For the ratio to hold on someone else's workload you would need a schema big enough that pasting it dominates the prompt, a query pattern repeated often enough to be worth wrapping in a tool, and a server whose responses are genuinely bounded. Tool design, restricted schemas and response limits decide that last one [13].

In my view the permission split is the first thing to copy, because it outlives whichever model you are using this quarter. The example in the post is specific: the agent may run npm test and may not run rm -rf, may read the database and may not write to it [7]. On the difference between a basic and a mature harness he is blunt: in the basic one you trust, in the mature one you audit [10].

Before the shared harness, his team's four frontend repositories were each configured differently, and pull requests went back and forth over the inconsistency [17]. Shared configuration is a reasonable fix for a review-queue problem.

The fifth section of the post is a checklist to run before granting autonomy, and the text breaks off before the items [19]. Model routing gets one line of policy: simple tasks to the cheap model, complex tasks to the expensive one, decided by the harness and not at each prompt [14].

What to watch

  • Whether the checklist for granting autonomy gets published.
  • Whether the 50k-to-15k reduction is reproduced across runs with the model and task fixed.
  • Whether the catalogue of around 80 steerings stays curated as the four frontend repositories change.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories