A flipped UTC offset sign passed all 680 of tomlkit's conformance tests

The green comes from one line in the test file. The expected value for every datetime case is built with `tomlkit._utils.parse_rfc3339`, which is the function under test ^[4]. Flip the sign inside the parser and the expectation flips with it, so the assertion compares a wrong value against itself. The author's conclusion is that the suite cannot see any datetime bug at all ^[5]. For integers and strings the corpus carries its own expected values, and an equivalent break there was caught within seconds ^[6].

Count what is left. 680 of the 1,058 tests are generated from toml-test ^[1], leaving 378 written by hand ^[7]. The generated cases are 64 percent of the suite ^[8], and on the datetime paths the hand-written remainder is what can fail at all. The three that did were two in `tests/test_utils` and one in `tests/test_items` ^[3].

For this to transfer to your parser, one thing has to be true: the harness that adapts the external corpus has to build its expected values by calling code the test is supposed to judge. toml-test is independent for `int` and `str` ^[6]. A library that wants a rich datetime object has to turn the corpus string into one somehow, and tomlkit does it with `parse_rfc3339` ^[4]. The adapter between someone else's corpus and your types is where the oracle can become your own code.

"Either the behaviour is correct, or the test cannot detect the behaviour being wrong," the author wrote on dev.to ^[9]. The separation is observational: run against correct code and check the count line actually ran the tests, break the real file the tests import rather than a copy, confirm the failure lands on the assertion with the expected-versus-actual you care about, then restore from git ^[10]. An ImportError proves only that the file loads ^[11]. "A test counts as verification once it has been observed red for the right reason," the author wrote ^[12].

The habit came out of MCP-Bifrost, an MCP server that hands mechanical code edits to a cheap worker model and validates them through a chain of gates ^[13]. Two of its first fifteen tests printed their failures and never exited non-zero, so any exit-code-based runner would have counted them as passing forever ^[14]. Every test in the project since has been seen red under a deliberate break before being trusted, several hundred of them ^[15].

The counting rule is the part I would steal first. Proving N mutations takes at least N+2 suite runs, and the author reports that line is what stopped a smaller model from claiming verifications it had not run ^[17]. One caveat on blame: a survived mutation is not automatically a vacuous test, because `<` to `<=` changes nothing for `clamp(5, 5)` ^[18]. Past a couple of mutations there is a harness that edits the real file, runs your real test command, restores from a hash-checked backup (restoring from memory is how you find out what else you edited), and reports which tests noticed each mutation, which mutations nothing caught, and separately which only broke the plumbing ^[19].