Skip to content

Build1 publisher3 min readPublished

Running this contamination gate on git diffs zeroes out its literal check

A dev.to post proposes scoring how much of an agent-written test came from the patch that shipped with it. The literal half of that score depends on ast.parse succeeding, and the invocation it documents feeds the checker a diff.

The Engineer · Build desk

Illustration accompanying Running this contamination gate on git diffs zeroes out its literal check

What happened

  • A dev.to post argues that when an agent writes a patch and its assertions in one pass, a green build often shows only that the two files agreed on a private story, not that the public contract holds.
  • Its proposed script scores the test against the patch and marks the pair contaminated at 0.35 token overlap or 0.25 literal overlap, exiting non-zero as a review flag rather than an automatic revert.
  • The first failure pattern it names is a model emitting src/fee.py and tests/test_fee.py together with the same timeout, the same error substring and the same sentinel UUID in both files.
  • The invocation it documents writes two git diff outputs into /tmp and hands the checker those two files.
  • The author labels the artifact a proposed local gate that has not been run against a private corpus, and tells readers to treat it as a method, not a result.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Half of the check depends on its input being parseable Python, so the configuration a team copies decides whether the shared-sentinel case is caught at all.
  • cost The proposal ships no code for re-deriving the oracle without the patch in view, and doing it means a second agent session for every change that touches behavior.
  • capability Where a reviewer previously had a pass or a fail, the report hands over up to 40 shared tokens and 40 shared literals to argue about by name.

The literal half of the score is the half that catches a shared sentinel UUID, and under the invocation the post prints, it cannot fire. `literals_from_ast()` parses the text it is handed with `ast.parse` and returns an empty set when that raises `SyntaxError` [9]. What it gets handed is `/tmp/prod.diff` and `/tmp/test.diff`, written by `git diff --unified=0 origin/main...HEAD` [10]. Diff text is not Python. Both literal sets come back empty, the intersection is empty, and `lit_score` evaluates to 0 over `max(1, 0)`, which is 0.0 [1]. Nothing reaches the 0.25 branch of the contamination test [7]. The report prints `literal_overlap: 0.0` and an empty `shared_literals` list. That output is indistinguishable from an uncontaminated pair.

The token side does survive the diff, because `tokens()` runs three regexes over raw text: identifiers of three characters or more, quoted strings of three or more, numbers of two digits or more, minus a fourteen-word stop list [12]. Both files handed to the gate are output from the same command, so whatever vocabulary the diff format contributes to each side lands in the intersection and counts as shared [2]. The denominator is the test's token count [6]. Scaffolding in the test diff lowers the score; vocabulary shared by both sides raises it.

Reading the two files instead of the two diffs fixes the AST and reintroduces the problem the author was avoiding. Whole-repo token sets are dominated by project vocabulary, and the post says that inflates the score and hides the leak [11]. At file scope the dilution is milder than at repo scope, but it is still imports, test framework names, and the public names the post tells you to expect, `compute_fee` and `FeeRequest` [13].

The second failure pattern was never going to show up in either score. "The production function grows an extra branch that exists only to satisfy a comment in the test," the post says, and coverage rises while behavior for real callers does not [4]. For that one it prescribes literal mutation plus an oracle session that never sees the patch [5]. The mutation helper appends `_x` to string constants of four characters or more and adds 1 to integers with absolute value of 10 or more [15]; if the suite still passes, the assertion was never bound to that value [16]. Constants outside those guards go through untouched, so an assertion pinned to 0, to 1, or to a three-character status string is not exercised by it [3].

Rewriting the oracle in a session that never sees the patch is the one prescription the post makes and does not implement [17]. The post is explicit about the status of the code that is there: a proposed local gate, not a published benchmark, not run against a private corpus, with the instruction "Label it as a method, not a result" [14]. The mutation helper carries the same warning in its header comment [18].

What to watch

  • A run of the gate against a real corpus, with flag rates, would show whether 0.35 and 0.25 separate private leakage from expected public API names.
  • A revision that reads the two file versions at their blob hashes instead of the diff text would restore the literal check without pulling in whole-repo vocabulary.
  • Whether anyone publishes the cost of a second, patch-blind oracle session per change in CI minutes and agent tokens.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories