Topic
Using language models to write unit tests for existing code, and the quality limits of the resulting suites.
A dev.to post-mortem swapped 40 percent of an end-to-end regression suite for LLM-written Playwright tests over six months. Its two published numbers, 85 percent passing first run and 30 percent flaky in month two, count different tests.
Reality
A dev.to how-to proposes four CI gates for LLM-written tests: execution, coverage, mutation, drift. Read the sample configs line by line and two of them would let through a test that asserts nothing.
Reality
Counting how many wrong implementations a test suite rejects is only as honest as the list of wrong implementations, and on a small Python order filter mutmut generated five candidates without the one that hid the regression.
Reality
A reader pointed out that flipping < to > leaves source size and whole-second mtime unchanged, so Python can import the original bytecode and log the mutant as surviving. The unparseable-garbage canary changes the file size, invalidates the cache and reports success.
Reality
One passing assertion on a three-line function covers 47% of its lines and catches 2 of 21 injected faults, which is the shape generated tests get accused of, though the harness measuring it needed eight fixes first.
Reality
A dev.to post walks through a Rails action that authenticates the caller and then loads any invoice by id. The defect is an absence, and the model's own tests confirm it.
Reality
A hand-written mutation survived all 20 generated pytest tests on a small CSV parser. Coverage counted lines executed; nothing in the suite checked the behaviour.
Reality