Build1 publisher3 min readPublished
Recording the score contract from the parent commit keeps the agent out of its own scorer
A dev.to proposal freezes fixture bytes, runner config and two test floors at the parent SHA, then denies the coding agent write access to the file holding them. Its content hashes hold up better than the path denylist it prints.
The Engineer · Build desk
What happened
- A dev.to post argues that a green CI job on an agent branch is only a claim that the tree the agent left behind still exits zero, and that the claim breaks once the agent can edit the suite.
- Its fix is a single contract file binding the parent SHA, fixture bytes, runner config, property budget, assertion floor, collected-test floor and time-bounded flake leases, with the agent denied write access to it.
- The recorder runs against the parent SHA before the agent starts, and after the patch returns the gate rejects any diff touching the contract, the scorer, fixtures or runner config.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- contradiction The post tells you to deny paths and then says path denylists fail; its own regex demonstrates the second point. The content hashes end up as the enforcing part of the design.
- constraint Byte equality on fixtures means an ordinary golden refresh cannot come from inside the agent loop; someone has to record a new contract from a parent tree first.
- decision Floors copied from the parent lock in whatever the existing suite already tolerates, so a team adopting this has to decide separately whether to raise them by hand.
- capability A dated lease turns a silenced test from an open-ended skip into a build that fails on a known day, and the renewal sits with a human.
The script meant to keep the agent out does not cover the directory it names. `deny_score_paths.sh` sets `DENY_REGEX` to `^(score_contract\.json|tools/score_lib\.py|tests/fixtures/|pytest\.ini|pyproject\.toml|conftest\.py|.*/conftest\.py)$` and greps the output of `git diff --name-only` [6]. The anchors wrap the whole alternation, so each branch has to match a complete line, and `git diff` prints `tests/fixtures/cassette.json`. The `tests/fixtures/` branch matches that literal string and nothing else, so a rewritten golden walks through the grep [7].
The post says why that layer is the wrong one to trust: "Path denylists fail when the agent adds a sibling file the loader prefers." [8] The hashes are the part that holds. `hash_tree` sha256s every file under the fixture root in sorted order, and `hash_runner_config` sha256s `pytest.ini`, `pyproject.toml` and `conftest.py` at the repo root plus every `conftest.py` below it [9]. Both maps go on the contract, the scorer recomputes them after checkout and requires equality, so a blank line added to a golden JSON is a miss. According to the author, that is intended [10].
Floors, by contrast, are weaker than the hashes, and the code shows why. A test name is a weak identity and a coverage percentage is weaker, because both move when the agent edits the suite instead of the code under test [19]. `count_asserts` walks `tree.rglob("test_*.py")`, parses each file with `ast`, and counts against a set of five `unittest` attribute names [11]. It also records the floor from the parent tree and checks the branch, since the post is explicit that the numbers come from the parent commit and never from the agent's branch [4]. A repo that names its files `*_test.py` therefore records an assertion floor of zero and then clears it [12].
The sample contract ships `assert_floor` 0 and `collect_floor` 0 next to a property budget of 100 examples and a 30 second deadline [13]. Zero floors admit a branch with zero asserts and zero collected tests, so the gate is inert until the recorder writes real parent numbers into it [14]. The post names the ceiling on this too: floors copy the parent, and if the parent suite is already tautological the contract preserves that weakness [15].
Leases are the piece I would adopt first. Each carries an RFC 3339 `expires_at`, the scorer may deselect a nodeid only while the lease is valid, and an expired lease is a hard failure instead of a quiet re-enable [17]. The alternative, in the author's words: "A skip with no expiry is a deleted test with better manners." [16] Proposed policy is 72 hours, one reason string, and no renewals from the agent [18].
None of this is measured. The post labels its numbers a proposal and its lease policy "Proposed lease policy, not a measured SLA" [18], and never gets to an incident or a failure rate. For the design to pay, your agents have to touch test surfaces often enough that a parent checkout, a clean runner and a denied write path cost less than reviewing the diffs by hand. The gates do not claim more than cardinality: collection count is the cheapest cardinality gate, and assertion count is a blunt density gate. Neither proves correctness [20].
What to watch
- A measured catch rate for the collected-test and assertion floors on real agent branches. The post has none.
- A recorder that hashes every file the test loader can read for config, not just the three names in CONFIG_CANDIDATES.
- A corrected DENY_REGEX, or a decision to drop the path script and rely on the hash maps alone.