Build1 distinct publisher2 min readPublished
One developer disabled each fail() call in his own engine and re-ran the suite once per line. The 36/36 green baseline measured whether gates bite on dirty input rather than whether any test reaches them.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The two numbers answer different questions, and both are honest. "36/36 biting mutants" reads as: for every mutant in the corpus, something went red [3]. It does not say which guard line bit, and it says nothing about the lines no mutant ever reached. Deletion is the only probe that separates those cases [1].
Arithmetic closes some of the gap before you run anything. Thirty-six mutants cannot be paired one-to-one with 61 guard lines, so at least 25 lines have no mutant of their own [4]. The measured result was worse than the pigeonhole floor: 40 unmeasured, leaving 21 lines whose presence any test can distinguish from their absence [2][2].
For 66% to transfer to your repository, the ratio would have to be a property of guards in general rather than of this guard set against this mutant corpus, and it is not: both come from one author's engine, measured in one release he already suspected [2][5]. Widen the operator set your mutants are generated from and the rate moves. Write guards for input classes your fixtures never produce and it moves the other way. That second case is the Blind Gate in the post's own taxonomy, alongside the gate that fell out of CI and the gate whose internal errors are swallowed by a soft error flag: it runs, it exits 0, and the input required to trigger its logic is never produced [6]. So the careful carry into review is narrower than the author's line about a comment pretending to be a guard [15]. Forty lines are unverified. That is not the same as wrong, and it is still not evidence.
The harness is linear in guard count. Sixty-one lines means 61 sabotage runs plus the baseline, so 62 full suite executions [3].
Two design rules keep it survivable. Each sabotage works on a temporary copy, so the working source file hafiza.py is never edited in place [13]. And the finder is an AST parse rather than a regex, because a regex silently misses multi-line calls, which would make the measurement tool blind in exactly the way it is trying to detect [12].
The detail I would steal is the encoding one. The hukum_kapisi.py gate matches explicit ASCII character patterns instead of UTF-8 dashes, because when Python's errors="replace" handler fires, a dash can print as "?", and a gate matching unicode dashes would be broken by the same encoding failure it exists to catch [14]. The other half of the proof is easier and gets skipped anyway: a gate also has to stay silent on clean input, or you have kept a false-positive generator with good manners [9].
Ranked by verification strength, evidence, and original report placement.
When the measurement finished across 61 protection lines in the engine, 40 of them were completely unmeasured by any test, a 66% blind spot rate.
The author wrote a script that parses the engine's source code, finds every fail() assertion call, disables them one by one, and re-runs the entire mutation test suite for each line, asking whether any test notices the deleted guard.
The baseline test suite run beside the sabotage measurement reported 36/36 biting mutants, a fully green result.
The script's exit codes are documented as 0 for every fail() covered with no blind spots, 1 for at least one uncovered fail() (annotated as expected in v2.4.1), and 2 for unmeasurable or setup error.
The author expected to find a few gaps and wrote that expectation directly into the script's exit codes, saying he knew he had blind spots but did not know how many.
The post classifies three ways a quality gate dies, all of which look green from outside: the Dead Gate that is no longer invoked by CI, the Environmentally Broken Gate whose internal errors are caught under a soft error flag, and the Blind Gate that runs, exits 0, and measures nothing because the input needed to trigger its logic is never produced.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 27, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
An AI test suite hit 94% coverage and missed the one branch that mattered1 distinct publisher
build
Force the tool call, then hand Lightsail a long-lived key1 distinct publisher
build
A cache hit is a quota refund: semantic caching with trigrams and no vector database1 distinct publisher
build
Stop timing your GraphQL tests and start counting loader calls1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single self-reported run, internally consistent but unreproducible
All numbers come from one dev.to post by the developer who wrote both the engine and the measuring tool. Method is described in enough detail to be re-implemented (AST call-site discovery, sabotage of temporary copies, compile() gating) and the arithmetic is internally consistent (40/61 = 65.6%; 36 mutants cannot pair with 61 guard lines). But there is no repository link, no released sabotaj.py, no raw results table (the per-gate section is truncated), and no independent replication, so the headline 66% figure cannot be checked.
No adoption signal beyond the author's own repository
The single disclosed use is the author running his own script against his own file. There is no release, package, license, star count, CI integration by others, or any third-party mention of the technique or tool in the supplied material, so adoption cannot be scored without inventing facts.
Mildly overstated generalization on an n=1 measurement
The headline framing ('40 of 61 unmeasured by any test', 'almost nobody writes tests for the third') invites readers to treat a one-repository result as an industry-wide pattern, and the technique is presented as Level 2 practice without cost, scaling or false-positive analysis. The overstatement is small rather than large because the author is candid about scope and motive: he pre-registered his expectation of gaps in exit code 1, deliberately classifies broken sabotages as UNMEASURABLE rather than UNCOVERED to avoid inflating the metric, and the arithmetic he reports holds up.
Personal-visibility incentive, no disclosed commercial stake
The author is assessing his own engine with his own tool and publishing the result on a developer-audience platform, so there is a reputational and audience-building incentive to present the method as novel ('Level 2') and the finding as striking. Countervailing: no product, pricing, sponsor, employer, or license is being promoted in the supplied text, and the self-critical framing (the tool that finds blind spots had one) works against a pure promotion read.
Method credible, magnitude unverified
Confidence is moderate-low: the described mechanism is technically coherent and the reported figures are mutually consistent, which supports believing the technique works as described. It does not support confidence in the magnitude or generality of the 66% blind-spot rate, given one publisher, one self-reporting author, one codebase, no released artifact, and a truncated results section.