Build1 publisher3 min readPublished
Classifying test files by glob hands the merge bit to whoever owns tests/oracles
A dev.to proposal sorts a repo's tests into human oracles, seeded property checks and quarantined flakes, then requires all three classes to agree before a merge. The classifier it ships checks paths and expiry dates.
The Engineer · Build desk
What happened
- A dev.to post proposes that a merge go green only when three test classes agree independently: human-owned oracles still failing known-bad fixtures, property checks passing on recorded seeds, and flaky tests excluded.
- Files the classifier cannot place default to untrusted, which means they may run and can fail the job, but their passes never count toward the merge bit.
- The classifier runs before pytest, prints a JSON inventory, and returns exit code 2 when a freeze has expired or 3 when no oracle tests were classified at all.
- The author presents the manifest, freeze file and classifier as a proposal to drop next to pytest, not a production SLA, with paths and markers left as local convention.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint ANDing the classes caps the gate at the human-owned oracle directory: no oracle files, no merge. A team adopting this has to produce known-bad fixtures at the rate its agent produces patches.
- contradiction The manifest promises human ownership and accepted invariants, but the code checks globs and dates. Those two guarantees have to live in CODEOWNERS or in a reviewer's head, not in the script.
- cost Quarantine turns into standing work. Every freeze carries an expiry date, and the day it passes, CI stops the merge until someone triages the flake.
- decision Directory layout becomes the trust boundary, so a team that keeps tests beside its source has to move files before the gate can classify anything.
Read `classify_file` before adopting any of this. It walks the oracle and property classes in order, fnmatches the relative path against each class's globs, and returns on the first hit; only if nothing matches does it compare the file stem against the active freeze ids, and whatever is left comes back untrusted [15]. So the freeze list cannot reach a file under `tests/oracles/` [23]. A timing-dependent oracle keeps its vote in the class that decides the merge.
The pattern `tests/oracles/**/*.py` [9] needs a literal separator after the `**`, so `tests/oracles/test_totals.py` does not match it and falls through to untrusted [31]. A repo that keeps oracles directly in that directory classifies zero of them and the script exits 3 on the first run [32]. That is at least failing in the right direction.
The manifest also declares markers for each class, `oracle`, `property` and `flake` [9][10][11], and the classifier never reads them. A test marked `oracle` living anywhere else classifies as untrusted, and an untrusted test can only fail the job [24][4]. `require_human_owner` and `require_accepted_invariant` are both set true in the manifest, and the shipped script checks neither [26].
The script enforces less than the three-class rule it argues for. The post publishes three files, the manifest, the freeze file and the classifier [29], and the runner that ANDs the class verdicts is not one of them [30]. The classifier prints the inventory, exits 2 with "expired freezes must be triaged before merge", and exits 3 with "no oracle tests classified; fail closed" [16]. It treats zero property tests as fine [25].
One manifest rule asks whether a test can detect a fault at all: `oracle_must_fail_known_bad: true`, with `known_bad_dir` pointing at `tests/oracles/known_bad` [8]. Nothing else in the manifest makes that demand [28]. It costs a maintained fixture for every fault class you care about, and it has to be human work, because the proposal bars the agent from adding oracle files in the same change that edits production code [5]. Agent-written property tests need a human to accept the invariant text and the seed policy first [6].
The scan is `(root / "tests").rglob("test_*.py")` [17]. Fail-closed therefore covers exactly the files under `tests/` whose names begin with `test_` [27]. Tests kept beside source, and fixtures without the prefix, never enter the inventory to be classified.
The sample freeze on `test_retry_backoff_under_load` runs from 2026-09-01 to 2026-09-29, with the reason "timing depends on shared runner load" [12]. That is 28 days [22]. `expired()` returns true when the expiry date is on or before today [18], so the gate stops on the 29th itself. The freeze file's comment gives the point of the expiry: a freeze should not be able to hide a permanent race [13].
For any of this to transfer, exit codes 2 and 3 have to block the branch. Ownership of `tests/oracles/` has to be enforced somewhere outside the script, and a person has to keep the known-bad fixtures current. Without that, what you get is a JSON census of test files by class. That is worth having on its own: it counts how many of your tests a human owns. "Agents are good at volume," the post says [21].
What to watch
- Whether a runner appears that actually ANDs the three class verdicts and enforces oracle_must_fail_known_bad against the known_bad directory.
- Whether classification moves from path globs to pytest markers, so a test's class travels with the file instead of the directory.
- Whether the freeze expiry policy survives contact with CI once a date like 2026-09-29 lapses and the classifier starts returning 2.