Skip to content

Build1 publisher2 min readPublished Updated

A must-reject lockfile blocks the merge when a frozen bad input starts returning 200

A dev.to proposal freezes each forbidden input as a hashed fixture next to the exception class or HTTP status it must still return, and keeps that file on a path the agent's own patch is not allowed to edit.

The Engineer · Build desk

Illustration accompanying A must-reject lockfile blocks the merge when a frozen bad input starts returning 200

What happened

  • The author's premise is that agent patches often go green by making invalid input succeed, which he calls not a fix but a silent spec change.
  • The failure he wants caught is not a red build but a forbidden case that now returns 200, Ok or a parsed object while the existing tests still pass.
  • Each corpus entry is a frozen input plus the exception class, status or error code that must still occur, and the gate fails if that input starts succeeding.
  • The corpus is meant to be human-owned: the agent may add candidate tests, but it must not rewrite the reject list in the same change.
  • The proposed layout is the corpus as JSON beside a small runner, with the lockfile treated as the spec and the runner kept mechanical.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Under the proposed path rule, every new forbidden case needs a reviewer's sign-off before it lands, so the reject list grows at review speed and somebody owns that queue.
  • exposure Teams copying the printed runner make exception names the contract, so an ordinary rename or a new subclass trips the gate. Nothing regressed; someone still has to triage the trip.
  • constraint Expectations cannot live where the agent can edit them, which forces the reject data out of the test tree and into a second artefact maintained separately from the suite it guards.
  • decision No measured run is published, so the first team to wire this into CI is producing the evidence: count how many of their own agent patches trip it.

Handling one lockfile row means importing a target and calling it. `load_target` splits the dotted path on the last dot, imports the module and returns the named attribute [14]. `expect_exception` then calls `fn(**payload)`, and if that call returns at all, the detail is "forbidden success: call returned" [15]. The fixture has to be a mapping of keyword argument names for that splat to work, and the example fixture is flat: `{"email": "not-an-email", "tenant": "t-1"}`, written with sorted keys and hashed with `hashlib.sha256` [19][27].

The class check compares strings. The printed line is `type(exc).__name__ != class_name` [16]. The code field narrows it, read off the exception as `getattr(exc, "code", None)` and compared only when the lockfile row declares a code [17]. Both example rows declare one: `email.invalid` on the `ValidationError` entry for `billing.api.create_user`, and `invoice.illegal_transition` on the 409 entry for `billing.api.transition_invoice` [8][9].

Per case, the work is one JSON fixture, one hash, and one lockfile row added by hand [10]. Each row in the example holds eight leaf values, and only `input_sha256` comes out of a tool, which leaves seven set by a person [23]. The article's placeholder digests are `c0ffee01` and `bada5502`, eight hex characters where a SHA-256 digest is sixty-four, which is presumably why the author tells you to replace them with real output from `sha256sum` [24][10]. The runner re-hashes each fixture on every run, so an edited fixture fails the gate [10].

The call happens in process, against whatever the dotted path imports [13][14]. A rejection performed upstream of that function, by gateway schema validation or an authz filter in middleware, never reaches the callable the gate invokes, so the gate does not see it [26]. Target selection follows from that. The author's advice is to pick the public functions an agent is allowed to touch and collect inputs production already rejects, naming invalid emails, closed-invoice transitions, cross-tenant ids and oversized payloads [20]. His stated reason for expecting gaps there is that existing tests assert success, leaving validators, authz checks and parse errors under-represented [2].

The runner listing is labeled "proposal, unexecuted in this article", and the published excerpt ends at the `expect_http` signature [22]. For this to transfer to another repo, the rejections that matter have to sit behind a function the runner can import, and the enumeration has to keep happening after the week someone set it up [20].

What to watch

  • The rest of the runner, including expect_http and what the two flake_policy values, never-skip and freeze-class-only, actually do.
  • A measured run on a real repository, showing how many agent patches trip the gate and how many of those trips turn out to be renamed exception classes.
  • Any tooling that makes the reviewer sign-off on corpus diffs machine-checkable instead of a review habit.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories