Nine bugs in a self-built mutation-testing harness, all favoring the builder's story

Mutation testing edits the source in small ways, flipping a comparison or changing a constant, then runs the test suite to see whether anything failed ^[4]. If the suite stays green, that is a fault the tests cannot detect ^[4]. The scoring step is where direction gets set. Flakiness can only move the number up here ^[10], because a mutant counts as caught when the suite fails, so any spurious failure counts as a catch, and detection was the number being maximised ^[10].

The import bug works the same way. Mutants that never execute are never caught, so the only reachable score is the floor, and a floor reads as a terrible test suite. That made the problem the author was solving look bigger ^[8]. No audit picked either direction; the definition of the metric fixed both before any result came back.

"You do not audit numbers. You audit numbers that bother you," the author wrote ^[2]. A disappointing result gets the setup checked and the logging added until the bug turns up; a good result triggers none of that and gets written up ^[3]. The author also wrote: "I knew this argument in the abstract before I started. It did not stop me from writing nine of them" ^[19].

Three of the nine inflated no headline number; they understated a baseline or a comparison arm ^[18]. The author counts them favourable anyway, on the grounds that a worse baseline makes the new work look better by comparison, and defines favourable as favourable to the story, not to one metric ^[18]. For a comparative claim that is the right definition. It also leaves half the tally where an outside reader cannot check it. On the strict count, six of the nine pushed the headline metric up ^[20]. Nine independent coin flips landing the same way happens once in 512 ^[21]. Six or more of nine landing the same way happens 130 times in 512, or about a quarter of the time ^[22].

The harness had switched off the repair mechanism it was measuring ^[14]. The extractor scanned only top-level functions, so a valid `unittest.TestCase` response was discarded as "no test found", and the retry loop then received a harness error instead of real pytest output ^[14].

The one formal control described in the post failed too: a pre-registered metric was not computable on dunder-dispatched code, and it reported a real-looking near-zero rate in place of undefined ^[16].

For the nine-of-nine result to transfer to someone else's harness, two conditions have to hold. The scoring rule has to convert incidental failures into credit ^[10], and one person has to be the only reader of the intermediate numbers ^[3]. Two of these nine bugs could not have landed the other way ^[8][10]. The remaining seven sat behind the author's own attention. That is the only filter the post describes ^[3].