Build1 publisher3 min readPublished
One planted requirement generated the six tests that certified it
Every gate in this AI codegen pipeline asks whether the code matches its instructions. The New Stack's account planted one bad instruction in a single unit and found nothing downstream that could fail it.
The Engineer · Build desk
What happened
- An account in The New Stack planted one bad requirement in a small unit, generated the implementation and tests from it, and watched six passing tests, a traceability gate and a clean run certify it.
- The feature that requirement belonged to existed to guarantee that one particular class of record never gets processed the way the planted sentence permits.
- The scoping document splits a feature into release groups and numbered units, roughly one per shippable slice, each with scope boundaries, dependencies, feature flags and given/when/then acceptance criteria.
- On one unit, a developer's question about a base class constructor revealed that the design document specified a call that would not compile, before any code existed to review.
- After the agent writes code, the change passes a local standards pass, a pull request with automated reviewers, a pipeline that blocks changes disagreeing with the spec, and automated QA, out to a weekly Thursday release.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Reviewing generated code harder cannot catch this class of defect, because the code is correct with respect to the instruction it was given.
- exposure The failure surfaces on the record population nobody manually tests, so it reaches production behind a green pipeline.
- decision Full call-path coverage stops being evidence that anyone weighed the requirement, so teams have to decide who reads requirements adversarially and at which gate that happens.
Generate the implementation and the tests from one sentence and they agree by construction. The tests encode the requirement, so they cannot dissent from it. The traceability gate compares code against spec, and on this unit it found a match [1]. It did what it was built to do.
The account prints the requirement with the domain stripped out: "If the classification lookup returns no determination, treat the record as permitted and proceed, so that an unavailable dependency doesn't block delivery" [2]. That is the shape of a requirement its author says he read earlier this year [3]. It names a real operational worry and offers a justification, and in a 40-page document a reviewer skims past it in two seconds, according to the account [20].
"Every control answers one question: Does the code conform to its instructions? The instruction itself never goes on trial," the author wrote [6].
Most of the described work happens before an agent sees anything. A feature starts as a recorded meeting holding every team the change touches, four or five groups for anything crossing a shared service, and when a contentious issue resolves somebody states the resolution out loud for the recording [7]. The transcript, not the requirements document that preceded it, generates the scoping document [8]. Product managers review that document, and once they sign off it becomes the source of truth while the initial requirements document becomes history [10]. It syncs to the issue tracker as one work item per unit [11]. A developer then generates a unit spec: named interfaces, method signatures, files to create or modify, an error-handling matrix, the step-by-step query flow, and a list of what the unit deliberately will not do [12]. Code is the fifth artifact in that chain [19].
The good engineering is in the routing rules. Open questions go back to a human instead of to whoever holds the keyboard [21], and the spec review checks the unit spec against the scoping document for uncovered requirements, quietly duplicated work and anything dropped in translation [14]. Coverage is all or nothing: the agent's output must cover the entire call path from entry point through the service layer, with unit and integration tests [15]. "Partial coverage of generated code is worse than zero coverage, because it falsely signals that a human thought about the untested paths," the author wrote [16].
This is one planted requirement in one unit, and the account does not say whether it went in before or after the product manager sign-off. The sign-off matters because it is the only step in the described chain aimed at a requirement itself; the spec review and the traceability gate each compare one artifact to another [14][1].
For the result to transfer to your pipeline, one condition has to hold: the tests have to come from the same artifact the code came from. If a human writes the acceptance tests from the feature's purpose, those tests can disagree with the requirement. If an agent generates both, six passes measure translation fidelity, and on this unit the sentence being translated was wrong [1].
What to watch
- Whether the account's two deferred unit-spec elements test a requirement against its outcome, or only add more conformance detail.
- A seeded-defect rate: how many planted requirements a pipeline like this catches out of how many inserted, and at which gate.
- Whether requirement-level adversarial review appears as its own tracked step with a named owner, rather than living inside product manager sign-off.