Skip to content

Build1 publisher3 min readPublished

This merge gate runs decode(encode(x)) before it trusts a green pytest run

A dev.to procedure keeps invertible pairs and normalizers in an in-repo registry, runs them as one isolated required check, and scans the same diff for assertions that got weaker while the suite stayed green.

The Engineer · Build desk

Illustration accompanying This merge gate runs decode(encode(x)) before it trusts a green pytest run

What happened

  • A dev.to procedure argues that a merge bar asking only whether pytest exited 0 accepts a codec that no longer round-trips, a normalizer that is no longer idempotent, and tests that passed because assertions got weaker.
  • The proposed gate runs three checks in order: round-trip on registered invertible pairs in the touched module, idempotency on registered normalizers, then an assertion-delta scan of test files in the same diff.
  • Functions that cannot be inverted, including hashing, HMAC and destructive deletes, are registered as one-way and kept off the round-trip list, and they get no faked inverse.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The gate only sees callables someone registered, and in the published example nine sample inputs are the whole oracle, so a new agent-written branch with no sample sails through it.
  • capability With the scanner required, an agent can no longer buy a green run by loosening an assertion inside the same patch, at least for the renames the table covers.
  • cost Someone now owns a separate required CI job and its sample set, and when it drags the prescribed fix is trimming samples. Trimming samples shrinks the oracle.
  • decision Reviewers have to start reading the registry diff on every agent PR, because that file is the only place a dropped pair shows up.

Read the test file before the argument. The round-trip case deep-copies each sample, computes `backward(forward(sample))`, and asserts the result equals the copy [9]. It then asserts the sample itself is unchanged, which fails when the encoder edited its input in place [9]. The idempotence case runs the function once, deep-copies that output, runs the function again on the copy, and compares the two [10].

The registry is the constraint. `RoundTrip` and `Idempotent` are frozen dataclasses carrying a name, the callables, and a tuple of samples [22]. The published example registers one pair, `record_json`, with four samples, and one normalizer, `normalize_path`, with five [11][12]. Nine inputs decide whether the gate fires [13]. "The samples are the contract," the dev.to post says [14], and an agent that adds a code path without adding a sample has extended the implementation while the oracle stays where it was [15].

The second route to green shows up in the test diff. Timeouts inflate, `assertEqual` becomes `assertIsNotNone`, a `pytest.raises` block disappears, and the production half of the patch stays small [16]. So the third check parses the patch and counts oracles that got weaker, rejecting them even when the suite passes [6]. In the published excerpt of `tools/assert_delta.py`, the `WEAKEN` table maps `assertEqual` to `assertTrue`, `assertFalse`, `assertIsNotNone` and `assertIsNone`, and the text breaks off mid-entry at `assertLis` [17]. The write-up calls the scanner "a starting heuristic, not a complete semantics engine" and says to label it that way when you wire it into CI [18].

Running it costs one required check, isolated from the rest of the suite: `python -m pytest tests/test_invariants.py -q --tb=short` [19]. "If this file is slow, the problem is the registry, not pytest," the post says, and it tells you to shrink the samples and keep the module running [20].

The article does not report measurement. It states that it is a procedure and that it is not a field report, that its examples are labeled, and that no pass-rate, latency, or model ranking is claimed [21]. The case for the three checks therefore rests on the shape of the failure. An agent rewrites one direction of a pair, encode still emits bytes and decode still returns an object, per-direction unit tests stay green, and `decode(encode(x)) == x` stops holding [3]. Coverage climbs anyway, because the new branch ran once against a fixture the agent also wrote [4].

I would take that trade in a repo with codecs, schema migrations, feature-flag parsers or cache keys, which the post names as the places the pattern recurs [5]. The registry file is where the human reading has to happen. The procedure lets an agent propose new pairs and forbids it from silently dropping old ones [8].

What to watch

  • A team that runs the gate for a quarter and publishes how many agent PRs it actually blocked.
  • Whether the assertion-delta scanner grows pytest-native rules beyond renames of unittest methods.
  • Whether registries drift in practice: pairs an agent proposed that no human reviewer ever checked.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories