Build1 publisher3 min readPublished
One hand-written ninth candidate made the redundant assertion the only check that caught the bug
Counting how many wrong implementations a test suite rejects is only as honest as the list of wrong implementations, and on a small Python order filter mutmut generated five candidates without the one that hid the regression.
The Engineer · Build desk

What happened
- On a catalogue of eight wrong implementations of a small Python order filter, a repeated-status assertion rejected nothing that another check in the suite had not already rejected.
- The author then added a ninth implementation, and the repeated-status assertion was the only check in the suite that caught it.
- Vinh Nguyen ran mutmut against the same fixture and reported that the candidates it generated did not include the condition substitution that produces the regression.
- A reconstruction of a second commenter's eight-candidate catalogue went from three rejections with the original two checks to eight once four further checks were added.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision The executable form of the review question, count how many wrong implementations a suite rejects, hands reviewers a delete recommendation for any check scoring zero unique rejections, and that score depends on who wrote the candidate list.
- constraint Two suites that genuinely disagree about a live regression get identical scores when nothing in the generated set exercises it, so the number cannot be used to rank them.
- cost Making the audit sensitive to a stated requirement costs a hand-written wrong implementation per requirement, and the reviewer writes it, because the generator did not.
- exposure Teams that prune tests on mutation scores are exposed in the operator gaps, and a falsey equivalence between None and an empty list sits in one of them.
The regression sits in one condition. The correct filter checks `if statuses is None` and returns a copy of the orders, otherwise keeping orders whose status is in the list [8]. Write `if not statuses` instead and Python treats both None and the empty list as falsey, so an empty filter returns every order [9]. The fixture's first two tests, a default call and a paid-status selection, pass on both versions, and only the empty-list assertion tells them apart [10]. The filter's stated rules require an empty list to return nothing [7].
mutmut did not generate that substitution. On a rerun with CPython 3.14.6 and mutmut 3.7.0 [12], the operators applied to the correct function inverted the identity comparison, replaced `list(orders)` with `list(None)`, altered the `"status"` key in two ways, and inverted membership [15]. Several generated candidates raised exceptions, and the suite caught those [16]. The correct implementation scored 5/5 with two tests and 5/5 with three, and the regressed implementation also scored 5/5 with two [13]. Add the third test to the regressed version and there was no passing baseline to mutate, so mutmut returned no score at all [14].
The score described all five generated candidates correctly, and it rated the two suites the same because both rejected the entire set [17]. Hand-adding the truthiness candidate raised the catalogue from five to six [25], and the whole new distinction between the suites came from that one candidate, derived from the empty-filter requirement [18].
A second catalogue, described in the comments by howcani, covers the filter condition, ignored filtering, an incorrect comparison, list identity, output order, repeated selectors and changes to the input [20]. In the reconstruction, the empty-list check took the count to four, requiring a new list object took it to five, an order-sensitive check to seven, and a check that the caller's input was unchanged to eight [21]. The order-sensitive check bought two rejections; each of the others bought one [24]. The author is clear that this is not a blind replication: the published descriptions and counts informed the reconstruction, and the original files were unavailable [22].
The repeated-status assertion looked redundant because of one reconstructed candidate. It iterates over the requested statuses first and the orders second, so it can reorder the result and duplicate an order when a status appears twice [23]. An order probe catches it, and so does `assert f(ORDERS, ["paid", "paid"]) == [ORDERS[0]]` [23]. Two checks, one candidate, and the count credits whichever one you happen to keep. With the ninth implementation in the catalogue, the five checks that reached 8/8 reject 8 of 9 [26].
For 8/8 to mean a suite is finished, the catalogue would have to contain every wrong implementation the code will actually attract. For a 5/5 to transfer, the bug would have to sit inside the tool's operator set, and the swap between None and the empty list was not in that set here [15]. The runs are local, on a deliberately constructed Python fixture, and the author says they are not measurements of a coding agent or a replication of ExecCritic [5]; the post says the result "gives no estimate of how often mutation testing misses important distinctions in other code" [19]. On the method itself the author wrote: "The approach is useful. Its denominator still needs review." [4]
What to watch
- Whether mutmut adds an operator that substitutes a truthiness test for an identity comparison, which would fold this bug class into generated candidates.
- Publication of howcani's original eight implementation files, which would let the 3/8-to-8/8 progression be checked without a reconstruction.
- Whether anyone runs the same catalogue audit on code that was not built to demonstrate the gap.