3,845 tests, 94.22% coverage, and nine things the suite could not see

Six plus two is eight, and the audit returned nine ^[4][9][16]. The maintainers' account itemises the validate-then-discard group and the pair of hygiene tests; the ninth appears only in the opening list, as a field that would silently discard a legitimate value in the version people were running ^[3][19]. That is the one I would most want the detail on, because it is the only finding described as changing what a user gets.

The shape of the six is worth carrying back to your own validators. Code checks an argument, the check passes, and then the line below reaches for the argument again instead of for whatever the check produced ^[9]. Nothing is wrong until the value can answer differently between those two reads, at which point the check and the use are no longer discussing the same object ^[10]. Reaching that in Engrava required a caller already executing inside the same process, not access through a config file, a query string, or data arriving from anywhere ^[11]. The maintainers then went back to the published package rather than their own notes about what they had fixed, and concluded that no version on PyPI crossed a trust boundary: no advisory owed, no CVE, no patch to the 0.5 line ^[12].

Their framing is the absence of a boundary rather than a hole in one, since anyone running arbitrary code in your process can replace the guarded function or the standard library call underneath it ^[13]. A test asserting they had defended against that caller would be theatre, in their words ^[14]. They hardened all six regardless, on the grounds that the mechanism generalizes to callers who are less trusted, and that a guard which can be talked out of its own conclusion is wrong on its own terms ^[15].

Now the part that changes how you read the other 3,845 ^[1]. Two of the nine findings, 22 percent of what the audit returned, were defects in tests rather than in production code ^[20]. Both sat in the hygiene suite, the path that decides what memory gets archived and eventually deleted, and both passed over guards that were working correctly while detecting something else entirely ^[16]. One carried a docstring stating in plain English that a specific pin caused a row to be skipped; a different condition was doing the skipping, and the pin protection could be stripped at all three layers with the test still green ^[17]. It was written by people who understood the feature and read by people who understood it too ^[18]. The team says it would have called the suite solid the day before the run ^[23].

That is the case against the headline number. 94.22% leaves 5.78% of lines that coverage describes accurately ^[22]. Inside the remainder, executed and checked look identical in the report, because a test can run a guard, assert something true about the result, and stay green after you delete the guard ^[6][7]. Reading the test does not separate the two either; it tells you what its author believed ^[18]. Mutating the code it claims to protect does, and what survives is the finding ^[8]. The reassurance in this repository was that it held more test code than production code ^[2], and that quantity was measuring the wrong property the whole time.