A green demo proves the agent can pass once; the shippable number is the failure rate

A post published on dev.to under the Loop & Retry banner makes a narrow and useful argument about agent readiness: a demo establishes only that the agent can succeed once, on an input you selected, on the happy path, in front of an audience ^[1]. The number a ship decision actually requires is different, and according to the author most of it is predictable before release if you test the right things ^[2].

The framing worth stealing is that the demo and the deployment run the same code, and that four things differ, each one a failure source the demo structurally cannot show you ^[3]. Input distribution: the demo uses inputs you picked, production uses inputs users bring, and the tails (empty fields, wrong encodings, a forty-page document where you tested a paragraph, an unsupported language) are most of the real distribution rather than an edge of it ^[4]. Load: the demo runs one task at a time, so throttling, timeouts and resource exhaustion are invisible at N=1 by construction ^[5]. Adversarial and malformed input: injection riding in on tool output does not appear unless you supply it ^[6]. Duration and drift: the demo runs for a minute, production runs for months across model updates and upstream schema shifts, and the author cites an earlier postmortem in which a field moving from optional to required turned 40% of a queue poisonous overnight ^[7].

The illustrative failure rate in the piece is 8% on the real distribution ^[8]. Worth doing the arithmetic on that before treating it as tolerable: it is roughly one failure in every twelve or thirteen inputs ^[9]. At any meaningful volume, that is not a tail, it is a queue.

The first test the author runs before shipping, ordered by production failure predicted per hour of effort, is failure injection ^[10]. The harness supplies six faults to a tool call: timeout, HTTP 500, HTTP 429, malformed JSON, an empty result, and a response with the wrong schema ^[11][12]. The instruction is to assert on recovery rather than the happy path ^[13]. Each probe records four things: whether the agent recovered, its retry count, the total cost of the run, and whether it asked for help ^[14]. That last pair is the part most suites skip. The two rows the author flags as most important are malformed and wrong_schema, which test whether a poisoned tool result cascades, and 429 and 500, which test whether the agent retries a permanent error into a runaway bill ^[15]. If it retries malformed fifty times, or escalates nothing when it gives up, you have found a production incident inside a unit test ^[16].

The reason teams do not run this, per the author, is not cost. It is that the work requires deliberately breaking something you have just got working, which is the opposite of what a green demo makes you want to do ^[17]. The economic case is stated plainly: the cheapest failure to fix is the one caught in a harness, the most expensive is the one your users find ^[18].

Two things to watch. First, whether your own test suite asserts on retry counts, run cost and escalation behaviour, or only on correctness, because the source's matrix is useless without those columns. Second, the published list continues past failure injection into distribution testing, and the text available to me is cut off at item two, so treat this as one technique from a longer sequence rather than the whole harness ^[19].