Split the suite so flaky tests cannot gate an agent's patch

The distinct exit code is the first detail I would copy. The check hashes the scoring properties, the fixtures, the map and the ledger, and stores that digest as a CI artifact from main. It then scores the candidate patch against that digest; when the patch has changed any of those files, the post says to fail with its own exit code and not to fold that failure into "tests failed" ^[18]. Those are different failures with different owners. A broken property goes back to the agent. A changed fixture digest goes to whoever owns the corpus.

The same scan keeps `pytest.mark.skip`, `xfail` and `flaky` out of the scoring glob, because those marks are how an agent converts a blocker into a pass, according to the write-up ^[17]. Test names are a weak measure and skip markers are weaker; the stronger one it proposes is a property that still runs, on frozen inputs, with an unchanged fixture digest ^[20].

On enforcement, the post says the agent gets no write access to the ledger ^[7], and the committed `scoring_map.yaml` carries a `forbidden_to_agent` list covering `tests/scoring/**`, the ledger file and the map itself ^[14]. That list is a key in a file inside the repo, and the digest comparison runs in CI after the patch exists ^[18]. What is described detects an edit; it does not block one ^[22]. If your harness hands the agent a branch and a commit, the ledger is writable and the check reports the change on the next run.

The ledger schema is specified down to the field. `flake_class` is a closed set of five values, timing, order, network, rng and env, and a test nobody can assign to one of them is not frozen at all, it stays off the scoring glob ^[10]. `evidence_hash` is a hash of the last failing artifact, and unfreezing requires a new evidence hash from a stability window instead of a deleted skip ^[11]. The post also rejects an expiry field as the primary control, on the grounds that expiry without a replay just returns the flake to the scoreboard ^[12]. Its worked entry moves `tests/scoring/test_invoice_poll.py` into the diagnostic tree on 2026-09-21, owner payments, class timing, with the note "Passes alone; fails under xdist when poll window is 50ms." ^[13]

The suite has to split one class per file with no mixed files, and that mapping happens before the agent is allowed near production code ^[9]; a module holding a property check and a live client call is two files of preparatory work. You also need properties to begin with: universal checks over a pinned input corpus that do not import wall-clock time, the network or process-wide RNG ^[4]. A mostly example-based suite lands on the diagnostic side, where it may flake and never votes on merge ^[6], and the scoring surface it leaves behind is thin.

The strategy is presented without measured results. Its reference script is labelled "reference implementation, not a published benchmark" and loads `scoring_map.json` ^[19], while the map committed in the walkthrough is `scoring_map.yaml` ^[14]. Copy both verbatim and the check dies before it hashes anything ^[21].