Skip to content

Build1 publisher3 min readPublished Updated

The proposed fixture scorer accepts any non-empty override token as a human approval

A dev.to scorecard for agent-authored patches digests every file under testdata/ and replays recorded property seeds before pytest runs. The published fragments enforce less than the prose around them describes.

The Engineer · Build desk

What happened

  • A dev.to post argues a green unit-test job is no evidence an agent patch is safe, because an agent can quiet pytest by rewriting golden files, dropping hard generator seeds or renaming a flaky case.
  • Its scorecard scores the diff against three files the agent does not own: a digest lock over everything in testdata/, a property seed log, and a freeze of failure signatures.
  • Each seed log row carries a property id, a pytest node id, an examples floor and the seeds that once found a counterexample, and the job fails if a recorded seed no longer runs.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • contradiction Anyone copying the code inherits a weaker gate than the article describes: the presence of an environment variable stands in for matching it, and the examples floor only trips below 1.
  • decision Adopting the lock forces a decision about who can set FIXTURE_OVERRIDE, since the agent-only property rests entirely on the secret plumbing and appears nowhere in the scorer.
  • constraint Byte-exact digests push canonical formatting into whatever writes the fixtures, because otherwise a routine reformat spends a human override.
  • cost Replay cost grows with every counterexample a team keeps; three seeds on one property already means three pytest startups on each diff.

The scorer fragment decides pass or fail on one line: `ok = not missing and (not changed or bool(override))`, with `override` read from the environment variable the lock file names [14]. The prose above it says a changed digest requires `FIXTURE_OVERRIDE` to match a value injected by a human job, not by the agent workspace [13]. `bool()` compares nothing. Any non-empty string satisfies it [15]. The human-only property is not in the scorer at all; it lives wherever that variable gets set, and the published check would accept `FIXTURE_OVERRIDE=1`.

The replay driver has the same gap running the other way. Lowering `max_examples`, tightening a strategy or filtering the input that used to fail all leave pytest exiting 0 [17]. Its stated rule is that the job fails when `min_examples` falls [18], and the example log sets that floor at 200 for `invoice_total_non_negative` [19]. The code records a floor failure only when the value is less than 1 [20]. Rewrite 200 to 1 and the driver passes, because 1 < 1 is false [21]. Enforcing the written rule needs a comparison against the log on main. The fixture check already does that for new paths, allowing them only when main's lock does not name them, and recording them as added [12].

In the example lock, the `sha256` field is 32 hex characters, half the 64 a hexdigest returns [22]. It is an illustration, and it would not match anything the scorer computes.

The sequencing is the part I would take first. Digest before pytest, because a dirty lock makes every green result after it untrusted [10]. The case the post gives for locking bytes is that committed testdata is an API and changing bytes is a behavior change even when the assertions move in the same diff [7], while unit tests encode today's examples and not the distribution those examples came from, so coverage can rise as the awkward case leaves the repo [5]. Digests cover raw bytes with no pretty-printing first [11]. A formatter run over a golden JSON file therefore flips the digest, which the post calls intentional, putting canonicalization in the writer [16]. Job strictness is set the same way: a skip is a fail, and a missing file is a fail [6].

Running it costs one pytest process per recorded seed [20]. The example property carries three seeds, so a single invariant spends three interpreter startups per diff [23]. The post labels the `--hypothesis-seed` flag environment-specific and says the contract is the log, not the library [24].

Two of the three contracts have file formats here. The failure-signature freeze is named as the third and the text breaks off in the seed section before specifying it [3][25]. The author is explicit that the scripts are a proposed harness and not production measurements [4], and for a design sketch the weak point is enforcement: the code is looser than the rules written above it [15][21].

What to watch

  • Whether the author publishes the failure-signature freeze with a file format and a comparison rule, completing the third contract.
  • Whether the fragments gain a value comparison for FIXTURE_OVERRIDE and a diff of min_examples against the seed log on main.
  • Whether the replay contract survives a runner that does not take --hypothesis-seed, which the post already flags as environment-specific.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories