Killing four mutation testing frameworks mid-run leaves one dirty working tree

cosmic-ray does what the first generation of mutation tools did: write the mutant into the file under test, run the suite, put the original back ^[13]. Both halves appear in its own log on a clean run:

``` INFO:cosmic_ray.mutating:Applying mutation: path=calc/__init__.py, op=<...NumberReplacer object...>, occurrence=2 INFO:cosmic_ray.testing:Running test (timeout=60.0): python -m pytest -q tests ```

^[14]

The restore is code, and code needs a living process. The uninterrupted fixture run takes 71.58 seconds ^[15] and completes 51 jobs ^[19], about 1.4 seconds per job ^[27]. The first mutant is on disk less than half a second after start, and a kill delivered there leaves `calc/__init__.py` mutated ^[16].

"I expected four dirty trees. I got one," wrote the author of the report, published on dev.to with the suite at Megapixel99/assay-checks ^[8][30]. Three came back clean because nothing in the tree was ever dirty: mutmut mutates a copy under `mutants/`, Stryker a copy under `.stryker-tmp/`, and PIT mutates bytecode in memory without writing a mutated `.java` file at all ^[9]. Clean here means every file is byte-for-byte what it was ^[28]. "There is nothing to clean up only if you never put anything there," the author wrote ^[12]. A handler only runs if the process is alive to run it ^[29], and after SIGKILL nothing is ^[3].

The argument the suite was built to test was that a harness passing all seven of assay's properties, killed by a timeout it exceeded, leaves the tree mutated exactly as though it had none of them ^[5]. cosmic-ray never reaches the uncatchable signal. A plain SIGTERM to its process left the file mutated, and SIGTERM is what `timeout` and `subprocess.run(..., timeout=...)` send before they escalate ^[17]. Each of those seven properties is a way a harness can report success without having run anything ^[1], and SIGTERM sits inside them ^[18].

The signal has to arrive while the mutant is on disk, which on this fixture is a window a fraction of a second wide, on a fixture chosen to make that window as wide as possible ^[20]. The workspace also has to outlive the process, which is the ordinary case when the killer is a CI runner's kill step rather than a fresh container ^[4]. On a real codebase with a slow suite, the author describes the per-mutant window as proportionally narrower with far more mutants; the total exposure is the same, arriving in smaller pieces ^[21].

The most useful result in the writeup is a bug in the writeup's own tooling. The first table reported Stryker clean on everything; Stryker had crashed ^[22]. The image was `node:22-bookworm-slim`, which does not ship `procps`, and Stryker shells out to `ps` to reap its test runners ^[23]. It died with `spawn ps ENOENT` immediately after the dry run, mutated nothing, and left a spotless tree ^[24]. Four passing verdicts about a run that never happened is the evidence property assay audits other harnesses for ^[25]. Every framework now has to print a declared proof-of-work string in its baseline, or the whole row reports `NO-RUN` ^[26].