Build1 publisher3 min readPublished
Ten packages each ship a test that would declare the package unnecessary
An exit status of 0 covers both a suite that passed and a suite that never ran, so ten new verification tools each commit a control test that fails if the tool's own reason for existing has gone away.
The Engineer · Build desk
What happened
- Ten small verification and measurement tools reached PyPI and npm between 20 and 31 August, eight of them on the last day, each encoding one way a green check can mean nothing.
- In a Go tree with no test files, go test ./... prints [no test files] and exits 0; a JUnit report can show fifty tests with fifty of them skipped.
- The post stakes itself on a checkable property: every one of the ten repositories commits at least one test that would report the package's own premise wrong.
- Seven of the ten READMEs report a deliberate-mutation pass over their own source, ranging from five mutations in restore-verified to 193 in assay.
- In canfail, the mutation that re-enabled mtime restoration survived the pass, which showed that an earlier fix had done nothing.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The price of the rule is one control per premise plus a mutation run over your own source, and the maintainer pays it again every time the runtime underneath changes.
- constraint A gate that reads only the exit status cannot separate a suite that passed from one that never ran, so the detection has to move inside the suite itself.
- decision Once a guard carries a test that fails when its premise dies, a red build becomes a reason to consider deleting the guard instead of debugging it.
- capability Reading mutation survivors gives a maintainer a list of code that no test constrains, including paths only reachable behind a flag nobody exercises.
`didrun`'s README states the rule all ten packages share: a check must answer separately whether it ran, whether it failed, and whether the failure was the right one [12]. Collapse any two of those three, the post argues, and you get every defect in this family [13]. What follows from it is a test-writing instruction. You cannot tell whether a guard is capable of failing until you give it something it must fail on [14].
Three of the controls test the reason their package exists. `nondet`'s asserts that calling a function 20 times inside one interpreter finds no variation, and that `nondet` finds the variation anyway in fresh processes, where Python's per-interpreter hash randomisation actually varies [19]. The post spells out the failing direction: if in-process repetition ever catches it, the fresh processes are wasted cost and the test says so [20]. `restore-verified`'s control spawns a real child running an ordinary `try/finally` harness, kills it, and asserts the mutated file stayed mutated [21]. That package exists because `finally` does not run on SIGTERM, so a red control there would mean the runtime changed, not that the tool broke [22]. `ladderpin`'s control switches off its determinism gate, pins a function whose output depends on hash order, changes not one byte of the tree, and asserts that the next check reports a change anyway [23].
Survivors of the self-mutation runs carried more information than the catches. mtime invalidation has one-second granularity, so a tool that edits, runs and restores in milliseconds defeats it whichever way the flag is set; the guard that actually prevents the stale import is `PYTHONDONTWRITEBYTECODE`, and the README had to be corrected [26]. `zerocase` had two survivors on its first run, one of them a branch reachable only through `--allow-stale` that nothing exercised [27]. Three of the ten READMEs do not report a mutation pass [1].
All ten are small verification and measurement tools, each in Python or JavaScript or both, installable from PyPI and npm [3]. In the control table, eleven rows cover the ten packages, because `assay-checks` carries two unrelated questions in one binary and each needs its own control [18].
A control has to clear a specific bar. It must show the naive alternative failing where the tool succeeds, or fixtures forcing the tool's verdict in both directions; a test that the tool works does not count [16]. Version stamps matter if you want to reuse a measurement: the all-skipped pytest run that prints `2 skipped` and exits 0 was measured on pytest 9.1.1 [8]. In each of these cases nothing about the repository looks wrong: a lint rule disabled by a config merge is green forever [10][11].
On frequency, the argument is thinner than it is on engineering. The research codebase the ten were extracted from is not public [6], and the reported instances are two: a memory profiler that watched the wrong process for eight months, and a rate limiter that reported requests it never sent [5].
What to watch
- Whether a reader checks the ten repositories and finds one without a premise test, which by the post's own terms would falsify it there.
- Whether the three READMEs with no mutation pass get one, and what survives when they do.
- Whether the didrun rule turns up in other projects' suites or stays inside these ten packages.