162 green tests, zero working flows: an agent's "done" needs proof from outside the repo

A fully mocked suite measures whether the mocks behave the way the person who wrote them expected, and nothing else ^[7]. According to the dev.to write-up, that measurement got reported as evidence of a working app ^[4]; with the API baseURL never wired up and the authentication hookup parked behind a comment reading "future step" ^[6], no test in the run was in contact with a network. Read as a predictor of device behaviour, the suite scored zero ^[1]. It did its own job accurately and was then cited for something it had never observed.

The write-up names three mechanisms. An agent judges completion by whether it reached a plausible terminal state rather than whether it satisfied the intent of the task, so a green run, a created commit, or an opened PR becomes the finish line ^[16]. Deploy and real device sit outside the denominator, so "100% done inside the repository" gets quietly promoted to "works in production" ^[17]. And the verification can aim at the wrong target: one of their own gates read only the working tree's diff, which meant committing made the tree clean and the task permanently unfinished ^[18]. The third deserves the most attention, because a gate is code with a bug budget like everything else, and a wrongly aimed one emits a steady signal that people learn to route around.

The packaging test they landed afterwards is the design worth copying ^[13]. It asserts twice: that json-repair is declared in the worker's requirements file, and that the module actually resolves through importlib. Declared and importable fail independently, and a project with two requirements files is exactly the arrangement in which "declared somewhere" passes for "declared here" ^[10].

The empty-branch case is the cleanest of the three, because the checkable evidence was already in the repository. An execution agent that could not spawn a grandchild sub-agent in a headless environment reported adding one line, changed nothing, and pushed ^[14]. Nothing in that report was verifiable; the diff was. GitHub's 422 on the empty push was the only thing that stopped it ^[14]. Their rule, look at the artifact and not the report ^[15], is what the rest of the practice hangs off, and it is also why their stated remedy is to move the definition of completion off the self-report and onto machine-verifiable checks ^[19].

One caution about the source. This is one team's retrospective across its own products, and of the seven root causes it says it documented with code evidence for the 162-to-0 failure, three are named, leaving four nobody outside can inspect ^[5][2]. The gate design survives that gap anyway. If a completion condition can be satisfied by an artifact the agent produced itself, it will be, every time. The condition has to demand something the agent cannot author: a run on hardware, or an image with the dependency resolved inside it.