Trusting the model's finish_reason let GPT-3.5 Turbo report a login page as success

The line to read in Branch 0 is the exit condition. The loop is a `while (true)` that calls `client.chat.completions.create` and pushes the reply onto `messages`. It returns the moment `choice.finish_reason === "stop"` ^[5]. That field arrives on the model's own response, and nothing else in the loop looks at the browser ^[19]. The comment above the return says as much: "The model says it's done, so we believe it." ^[5]

The post lists six parts: a tool registry, a model, context management, guardrails, an agent loop and a verify step ^[8]. Only the last one runs after the model has declared itself finished ^[21]. Guardrails stop an action on the way out; the verify step checks the claim against the environment ^[1]. The author says the most common question he got while preparing his talk on this was whether the harness is just the agent loop ^[18]. His answer: the harness is everything around the loop, and can be a loop around your loop ^[9].

The definitions in circulation are subtractive. LangChain writes that "a harness is every piece of code, configuration, and execution logic that isn't the model itself" ^[12]. Birgitta Boeckeler on martinfowler.com calls it "everything in an AI agent except the model itself" ^[13]. The post prefers a job description, grounding a model you don't control in an environment you do, on the grounds that a job tells you what to build next ^[14]. Read that way, Claude Code is a model wrapped in tools, context compaction and limits ^[15].

The six-iteration figure is a report about one task in a scripted browser driven by one cheap model ^[6][3]. It transfers to your workload only if your task carries a check the harness can run for less than the work costs. That check is the equivalent of reading the page back to see whether the vote landed. On a task whose outcome only the model can see, the verify step has nothing to compare against and the run ends on `finish_reason` again ^[19]. You have to do the mapping from the six iterations to the six parts yourself ^[20]. The repository publishes one branch per step, so the diffs are inspectable ^[11].

Holding the prompt fixed is the discipline I would keep ^[7]. It forces every gain between branches to be attributed to code, which is the claim under test ^[10]. The author's reason for starting from a weak model: "If a harness can make a weak, cheap model do real work, it can make a strong one boringly reliable." ^[16]