Skip to content

Build Not yet confirmed elsewhere1 publisher 3 min read Published

878 tests, zero installs: what agent-built code checks and what nobody encoded

Vestibule's suite was green while pip install was broken. An AI reviewer caught a 390-second race in a design document, because someone had written the document.

The Engineer · Build desk

How we use AI Send a correction

What happened

  • Vestibule was built over two months mostly by four AI agents working through GitHub pull requests, with the author signing off at each gate.
  • After v0.2 shipped, the author ran the pipeline as a stranger would and found pip install failed entirely on a packaging conflict, with 483 tests green.
  • The same hour turned up a default model name that had never worked against the real SDK and an import that broke a whole package when an optional dependency was missing.

Why it matters

  • constraint A review agent can only inspect an artifact that exists, so any requirement with no written design or test, including packaging and first run, passes unexamined by construction.
  • decision For teams reading agent output, this reprices where the next unit of effort goes: hundreds of extra tests did not close the gap that one clean-install job did.
  • exposure Everything behind a mocked seam stays unverified until a stranger hits it on a clean machine, which means the person discovering it is a first-time user rather than a maintainer.
  • precedent Naming all three install-path bugs in the release notes sets a checkable standard other agent-built repositories will now be measured against.

The race condition was catchable because it existed in prose. One agent produced a design, another agent read it, and a roughly 390-second index-creation call sitting next to a 300-second staleness default was right there on the page [5]. Five rounds of that before any code was written [4]. The packaging conflict had no equivalent artifact. Nothing in the four-agent loop produced a document describing what happens when a stranger types pip install, so nothing reviewed it, and the defect survived into a release with 483 tests green behind it [6].

The three bugs found in that first cold hour are one bug in three costumes. Each is an assertion about something outside the working tree. A default model name is only wrong once a real SDK answers [7]. An optional dependency is only absent on a machine that never installed it [7]. A dependency resolver only conflicts when it actually resolves [6]. On the author's own telling, the suite proved the code agreed with itself across the same tree and the same mocked seams, and nothing in it ever touched a clean machine [8].

Here is the arithmetic the post does not do. The suite was at 483 when install was broken, and the project reports 878 at v0.3, so roughly 395 tests were added after the discovery [11]. None of them is the thing that fixed the class of failure. That was a single CI job which builds a clean virtualenv, installs for real, and runs the quickstart on every pull request [9]. One job covered a category that 483 tests had not entered, which is a reasonable way to price test counts as evidence of readiness.

The asymmetry is the part worth carrying to other projects. Concurrent index provisioning is genuinely hard, and it got five design reviews and a caught defect before implementation [4][5]. Packaging is not hard, and it got nothing, because nobody encoded it as a gate. Reviewer agents inherit the attention of whoever wrote their brief, and that attention followed the interesting problem. The default staleness threshold is a small monument to the same habit: at 300 seconds against a 390-second call, the claim lapses by 90 seconds every time creation is slow, in the shipped default, before anyone tunes anything [12].

This is one developer's account of his own project, published to promote it, and it should be read that way [1][2]. The checkable part is that all three install-path bugs are named in the release notes, and the project says at v0.3 what is still missing [9][10]. That is more than most repositories with a green badge offer. The transferable asset is not the framework. It is the observation that an agent pipeline reviews artifacts, and install, first run, and real-SDK behaviour produce no artifact unless someone makes one.

What to watch

  • Whether the clean-virtualenv install job stays enforced at v0.4 and later, or gets marked allowed-to-fail the first time it blocks a merge.
  • Whether an independent user reports the 60-second quickstart working on a clean machine, rather than the author.
  • Whether other agent-run repositories publish an install gate alongside their test counts, or keep quoting the test count alone.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence34
Adoption12
Hype gap+8
Incentives68
Confidence42
Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    Vestibule is an open-source Python framework for the RAG ingestion layer, covering stable document IDs, a state ledger, error classification and per-vertical governance, built over two months.

    Reported Supported Source: dev.to post by the project's authorView cited source
  2. [2]

    Most of the code was written by four AI agents (one wrote designs, one reviewed designs, one implemented, one reviewed code) through real GitHub pull requests, with the author signing off at every gate.

    Reported SupportedView cited source
  3. [3]

    The project reports twelve components, three releases and 878 tests.

    Reported SupportedView cited source

Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.

Add to ChatGPTAdd to Claude

Sources

1 independent publisher whose own reporting we read for this story.

  1. dev.to

    1 article · August 22, 2026

    483 tests passed, but Vestibule RAG framework wasn't installable — lessons from building with AI agents

Share your take

Let Clarity write the post for you.

Signed-in readers get a short post drafted on this story in the register they choose — narrative, analytical, or a direct position — editable to the last word before it goes anywhere. The share buttons at the top of this story work without an account.

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Entities

Loading related stories