Topic
A software testing technique that injects small deliberate code changes (mutants) to check whether existing tests fail, measuring test suite effectiveness.
Megapixel99's assay scan pairs duplicate functions whose outputs match on one fixed input ladder, a check it can apply to roughly a tenth of functions. A match means only that no probe split the pair, so the tool fails the run and hands it to a person.
Reality
zerocase fails a CI run when its JUnit, TAP, LCOV, Cobertura or ESLint report shows zero executed items, even if the command exited 0. It counts only what ran, so fifty skipped tests fail the gate and fifty failed tests are left to the exit code.
Reality
Mutation testing needs a function to mutate, so the YAML and Terraform that gate production go untested. canfail edits an anchored string in those files, runs your check and reports whether it failed for the reason you declared.
Reality
Filament Studio's developer stopped trusting his 1,800 passing tests in September and installed the package into a real Laravel app. The defects he logged share one shape: success reported, nothing done.
Reality
NFKD maps U+FF0C onto an ordinary comma, so the name 北京，上海 sanitizes down to a single comma and never trips a check for the empty string. The transaction imports with the right amount and a blank payee.
Reality
An exit status of 0 covers both a suite that passed and a suite that never ran, so ten new verification tools each commit a control test that fails if the tool's own reason for existing has gone away.
Reality
A conformance suite hashed a repository, killed mutmut, cosmic-ray, Stryker and PIT at the instant a source file went dirty, then hashed again. One of the four came back mutated, and a plain SIGTERM was enough to do it.
Reality
In tomlkit, the official TOML conformance corpus builds every expected datetime value by calling the parser under test, so parser and expectation drift together. Three hand-written tests caught the break.
Reality
A dev.to post proposes scoring how much of an agent-written test came from the patch that shipped with it. The literal half of that score depends on ast.parse succeeding, and the invocation it documents feeds the checker a diff.
Reality
A developer ran three change tasks in one Python and TypeScript product against two code-index tools, and counted a run only after the focused test passed and then failed again with the defect put back on purpose.
Reality
Salesforce Core failed globally on the second morning of Dreamforce 2026, and the standard Apex loop keeps its compiler, test runner and debugger inside the org. The only account of shipping through it comes from a toolmaker.
Reality
Prose in AGENTS.md did not stop it, so the project now enumerates agent skills from disk and runs a script that exits non-zero on any copy outside the canonical root, including an empty leftover directory.
Reality
A dev.to how-to proposes four CI gates for LLM-written tests: execution, coverage, mutation, drift. Read the sample configs line by line and two of them would let through a test that asserts nothing.
Reality
Counting how many wrong implementations a test suite rejects is only as honest as the list of wrong implementations, and on a small Python order filter mutmut generated five candidates without the one that hid the regression.
Reality
A rule can evaluate cleanly, appear in the rule list, report healthy, and still be incapable of going red. One team found four of those in its own stack, and the PromQL one turned on a missing keyword.
Reality
A dev.to workflow makes every recorded golden prove it can fail before a refactor starts. The gate script that enforces it drops any of its three hardcoded patterns the target file does not contain.
Reality
A dev.to post reports ten bugs in one measurement harness, every one of them flattering and none found by reading the code. The three checks that eventually caught them cost a few lines each to write.
Reality
DHSeaDev's write-up on the browser card game Prismwar shows a replay check passing ten of ten matches because the recorder and the playback player both skipped the AI's defence step, so the round trip compared a bug against a copy of itself.
Reality
Eleven passes and one failure on a footer change turned out to predate the diff. The test built tomorrow in UTC while the function counts days in America/Los_Angeles, so it goes red from midnight UTC to about 07:00.
Reality
A reader pointed out that flipping < to > leaves source size and whole-second mtime unchanged, so Python can import the original bytecode and log the mutant as surviving. The unparseable-garbage canary changes the file size, invalidates the cache and reports success.
Reality
Build · September 11, 2026 · 1 publisher
Build · September 11, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 7, 2026 · 1 publisher
Build · September 2, 2026 · 1 publisher
Build · September 1, 2026 · 1 publisher
Build · August 30, 2026 · 1 publisher
Build · August 28, 2026 · 1 publisher
Build · August 27, 2026 · 1 publisher
Build · August 24, 2026 · 1 publisher
Build · August 24, 2026 · 1 publisher
Build · August 24, 2026 · 1 publisher
Build · August 22, 2026 · 1 publisher
Build · August 21, 2026 · 1 publisher
Build · August 20, 2026 · 1 publisher
Build · August 16, 2026 · 1 publisher