Build1 publisher3 min readPublished
Running this contamination gate on git diffs zeroes out its literal check
A dev.to post proposes scoring how much of an agent-written test came from the patch that shipped with it. The literal half of that score depends on ast.parse succeeding, and the invocation it documents feeds the checker a diff.
The Engineer · Build desk

What happened
- A dev.to post argues that when an agent writes a patch and its assertions in one pass, a green build often shows only that the two files agreed on a private story, not that the public contract holds.
- Its proposed script scores the test against the patch and marks the pair contaminated at 0.35 token overlap or 0.25 literal overlap, exiting non-zero as a review flag rather than an automatic revert.
- The first failure pattern it names is a model emitting src/fee.py and tests/test_fee.py together with the same timeout, the same error substring and the same sentinel UUID in both files.
- The invocation it documents writes two git diff outputs into /tmp and hands the checker those two files.
- The author labels the artifact a proposed local gate that has not been run against a private corpus, and tells readers to treat it as a method, not a result.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Half of the check depends on its input being parseable Python, so the configuration a team copies decides whether the shared-sentinel case is caught at all.
- cost The proposal ships no code for re-deriving the oracle without the patch in view, and doing it means a second agent session for every change that touches behavior.
- capability Where a reviewer previously had a pass or a fail, the report hands over up to 40 shared tokens and 40 shared literals to argue about by name.
The literal half of the score is the half that catches a shared sentinel UUID, and under the invocation the post prints, it cannot fire. `literals_from_ast()` parses the text it is handed with `ast.parse` and returns an empty set when that raises `SyntaxError` [9]. What it gets handed is `/tmp/prod.diff` and `/tmp/test.diff`, written by `git diff --unified=0 origin/main...HEAD` [10]. Diff text is not Python. Both literal sets come back empty, the intersection is empty, and `lit_score` evaluates to 0 over `max(1, 0)`, which is 0.0 [1]. Nothing reaches the 0.25 branch of the contamination test [7]. The report prints `literal_overlap: 0.0` and an empty `shared_literals` list. That output is indistinguishable from an uncontaminated pair.
The token side does survive the diff, because `tokens()` runs three regexes over raw text: identifiers of three characters or more, quoted strings of three or more, numbers of two digits or more, minus a fourteen-word stop list [12]. Both files handed to the gate are output from the same command, so whatever vocabulary the diff format contributes to each side lands in the intersection and counts as shared [2]. The denominator is the test's token count [6]. Scaffolding in the test diff lowers the score; vocabulary shared by both sides raises it.
Reading the two files instead of the two diffs fixes the AST and reintroduces the problem the author was avoiding. Whole-repo token sets are dominated by project vocabulary, and the post says that inflates the score and hides the leak [11]. At file scope the dilution is milder than at repo scope, but it is still imports, test framework names, and the public names the post tells you to expect, `compute_fee` and `FeeRequest` [13].
The second failure pattern was never going to show up in either score. "The production function grows an extra branch that exists only to satisfy a comment in the test," the post says, and coverage rises while behavior for real callers does not [4]. For that one it prescribes literal mutation plus an oracle session that never sees the patch [5]. The mutation helper appends `_x` to string constants of four characters or more and adds 1 to integers with absolute value of 10 or more [15]; if the suite still passes, the assertion was never bound to that value [16]. Constants outside those guards go through untouched, so an assertion pinned to 0, to 1, or to a three-character status string is not exercised by it [3].
Rewriting the oracle in a session that never sees the patch is the one prescription the post makes and does not implement [17]. The post is explicit about the status of the code that is there: a proposed local gate, not a published benchmark, not run against a private corpus, with the instruction "Label it as a method, not a result" [14]. The mutation helper carries the same warning in its header comment [18].
What to watch
- A run of the gate against a real corpus, with flag rates, would show whether 0.35 and 0.25 separate private leakage from expected public API names.
- A revision that reads the two file versions at their blob hashes instead of the diff text would restore the literal check without pulling in whole-repo vocabulary.
- Whether anyone publishes the cost of a second, patch-blind oracle session per change in CI minutes and agent tokens.