878 tests, zero installs: what agent-built code checks and what nobody encoded
Vestibule's suite was green while pip install was broken. An AI reviewer caught a 390-second race in a design document, because someone had written the document.
The Engineer · Build desk
What happened
- Vestibule was built over two months mostly by four AI agents working through GitHub pull requests, with the author signing off at each gate.
- After v0.2 shipped, the author ran the pipeline as a stranger would and found pip install failed entirely on a packaging conflict, with 483 tests green.
- The same hour turned up a default model name that had never worked against the real SDK and an import that broke a whole package when an optional dependency was missing.
Why it matters
- constraint A review agent can only inspect an artifact that exists, so any requirement with no written design or test, including packaging and first run, passes unexamined by construction.
- decision For teams reading agent output, this reprices where the next unit of effort goes: hundreds of extra tests did not close the gap that one clean-install job did.
- exposure Everything behind a mocked seam stays unverified until a stranger hits it on a clean machine, which means the person discovering it is a first-time user rather than a maintainer.
- precedent Naming all three install-path bugs in the release notes sets a checkable standard other agent-built repositories will now be measured against.
The race condition was catchable because it existed in prose. One agent produced a design, another agent read it, and a roughly 390-second index-creation call sitting next to a 300-second staleness default was right there on the page [5]. Five rounds of that before any code was written [4]. The packaging conflict had no equivalent artifact. Nothing in the four-agent loop produced a document describing what happens when a stranger types pip install, so nothing reviewed it, and the defect survived into a release with 483 tests green behind it [6].
The three bugs found in that first cold hour are one bug in three costumes. Each is an assertion about something outside the working tree. A default model name is only wrong once a real SDK answers [7]. An optional dependency is only absent on a machine that never installed it [7]. A dependency resolver only conflicts when it actually resolves [6]. On the author's own telling, the suite proved the code agreed with itself across the same tree and the same mocked seams, and nothing in it ever touched a clean machine [8].
Here is the arithmetic the post does not do. The suite was at 483 when install was broken, and the project reports 878 at v0.3, so roughly 395 tests were added after the discovery [11]. None of them is the thing that fixed the class of failure. That was a single CI job which builds a clean virtualenv, installs for real, and runs the quickstart on every pull request [9]. One job covered a category that 483 tests had not entered, which is a reasonable way to price test counts as evidence of readiness.
The asymmetry is the part worth carrying to other projects. Concurrent index provisioning is genuinely hard, and it got five design reviews and a caught defect before implementation [4][5]. Packaging is not hard, and it got nothing, because nobody encoded it as a gate. Reviewer agents inherit the attention of whoever wrote their brief, and that attention followed the interesting problem. The default staleness threshold is a small monument to the same habit: at 300 seconds against a 390-second call, the claim lapses by 90 seconds every time creation is slow, in the shipped default, before anyone tunes anything [12].
This is one developer's account of his own project, published to promote it, and it should be read that way [1][2]. The checkable part is that all three install-path bugs are named in the release notes, and the project says at v0.3 what is still missing [9][10]. That is more than most repositories with a green badge offer. The transferable asset is not the framework. It is the observation that an agent pipeline reviews artifacts, and install, first run, and real-SDK behaviour produce no artifact unless someone makes one.
What to watch
- Whether the clean-virtualenv install job stays enforced at v0.4 and later, or gets marked allowed-to-fail the first time it blocks a merge.
- Whether an independent user reports the 60-second quickstart working on a clean machine, rather than the author.
- Whether other agent-run repositories publish an install gate alongside their test counts, or keep quoting the test count alone.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence34
- Adoption12
- Hype gap+8
- Incentives68
- Confidence42
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Vestibule is an open-source Python framework for the RAG ingestion layer, covering stable document IDs, a state ledger, error classification and per-vertical governance, built over two months.
- [2]
Most of the code was written by four AI agents (one wrote designs, one reviewed designs, one implemented, one reviewed code) through real GitHub pull requests, with the author signing off at every gate.
- [3]
The project reports twelve components, three releases and 878 tests.
- [4]
The design of the component that provisions vector indexes on first use was rejected and revised five times before any code existed.
- [5]
In the first design review round, the reviewer agent found a race condition in the default configuration: a worker inside a slow index-creation call (about 390 seconds with retries) would look stale against a 300-second default threshold, lose its claim to a waiting worker, and two workers would create the same index.
- [6]
After v0.2 shipped, the author wrote a quickstart script and ran the pipeline as a stranger would for the first time; pip install did not work at all because of a packaging conflict that made the framework uninstallable, while 483 tests were green.
- [7]
The same hour of real use turned up two further defects: a default model name that had never once worked against the real SDK, and an import that took down an entire package when an optional dependency was absent.
- [8]
The author's diagnosis was that the tests proved the code agreed with itself in the same working tree with the same mocked seams, and that nothing checked a clean machine, a real install or a real SDK.
- [9]
The lasting fix was a CI job that builds a clean virtualenv, performs a real install and runs the quickstart on every pull request; all three bugs are named in the release notes.
- [10]
The project is at v0.3 and the release notes state plainly what is missing.
- [11]
About 395 tests were added between the 483-test suite that was green while the framework was uninstallable and the 878 tests reported at v0.3.
- [12]
A 390-second creation call against a 300-second staleness default exceeds the threshold by 90 seconds, about 1.3 times the limit, so the claim lapses on every slow creation in the shipped default.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI Design ReviewFollow
- Agentic Software DevelopmentFollow
- Concurrency race conditionsFollow
- Packaging and Release EngineeringFollow
- RAG Ingestion InfrastructureFollow
- Test Coverage vs Real UseFollow