Five rewrites later, the LLM is out of the test loop and into the selectors

Variance, not price, is what makes an execution-loop agent unusable as a monitor. A health check exists to emit one bit that something downstream acts on, and the author reports that the same test could pass or fail depending on how the model read the page that day ^[7]. That adds a failure mode nobody can separate from the real one: a red check no longer distinguishes a broken cart from a model that read the page badly. Cost only decides how long you put up with it.

The cost is still worth doing the arithmetic on, because the author states the intended cadence plainly: cron, many times a day, forever, with "every hour" as the reference point ^[8]. Hourly is 24 executions a day and 8,760 a year for a single scenario ^[20]. Each one carried seconds of latency per step ^[7] and a metered API call, and in practice a handful of runs cost real money and hit rate limits almost immediately ^[6]. Generation of a full test in the later version costs a fraction of a cent ^[15], which puts per-run execution at least two orders of magnitude above per-test generation ^[21]. The scenario does not change between runs and the page usually does not either ^[9], so the model is being paid repeatedly to reach the conclusion it already reached.

Moving the model to generation time only works if the model can see the page cheaply, and that is the part of the write-up doing the real work. A production product page is around 2MB of HTML, tens of thousands of tokens of noise; the Playwright accessibility snapshot of the same page is about 2KB of JSON, 500 to 1,000 tokens, and it holds the buttons, links, inputs and their labels that a test actually touches ^[14]. That is roughly a thousandfold reduction in payload ^[19], and it is why Haiku is sufficient to write the test rather than a frontier model ^[15].

One snapshot is not enough, because the tree of the search page says nothing about the cart page ^[16]. The fix was a persistent browser over CDP with an agent orchestrator generating one step, executing it live, snapshotting the new state, then writing the next, with the assembled file gated by pytest before it is saved ^[17]. Note what that is: the same interpretive work as the execution loop, run once, with a passing test as the artefact instead of a log line.

The earlier bookmarklet version is the counterexample that explains the ordering. Human clicking made generation very cheap at roughly 600 tokens ^[11], but the page loaded without proper JavaScript execution, so client-rendered elements were invisible, and hand-clicking every element restored exactly the manual labour the project existed to remove ^[12].

The self-healing payoff is asserted, and the supplied text cuts off at the point where the three-file test folder is described ^[18][22], so the repair path is not verifiable here. What the rest of the account already supports is the shape of the bill: model spend tracks how often the storefront changes, not how often you check it.