Topic
Tests whose pass/fail result varies across runs due to randomness, timing, or small sample sizes, undermining confidence in CI results.
A dev.to post names five habits that rot a Playwright suite, starting with waitForTimeout used to patch a race condition. Working back from its one-bad-run-in-four figure shows how little per-test failure that takes.
Reality
Across twelve ORBITAL tasks, eight failed a browser check at least once, and the verifier's own triage marked six of them automation bugs. Those six were cleared by override, two of them with per-case approval.
Reality
A dev.to proposal freezes fixture bytes, runner config and two test floors at the parent SHA, then denies the coding agent write access to the file holding them. Its content hashes hold up better than the path denylist it prints.
Reality
A dev.to write-up sorts every test path into four classes and hashes the scoring ones on main, so a patch that retouches a fixture fails with a different exit code than a patch that breaks a property.
Reality
The obvious determinism check calls a function twice in one process, where the hash seed never changes. nondet re-runs each function against a fixed input ladder in fresh workers, and refuses to probe I/O.
Reality
A dev.to engineer's runbook treats flaky microservice tests as incidents to be measured and isolated before anyone reaches for a fix. Its fifty-run reproduction loop is what decides which flakes a team can diagnose at all.
Reality
A dev.to writeup puts a 90-spec Cypress-to-Playwright migration at four working days with Claude Code. Most of the design work was 90 minutes of hand translation and one rule telling the agent when to stop.
Reality
Six months in, a Playwright and Flutter test engineer reports that scaffolding generates fine from a plain-language bug report plus existing page objects, and that every generated assertion gets rewritten by hand.
Reality
One developer's four clean runs against livekit-client 2.22.3 all read the track dimensions after the preview appeared. Reading there lands outside the roughly 100 ms window in which a portrait iPhone claims 1920 by 1080.
Reality
A dev.to post gives every failed coding-agent run one of four labels and scores skill only over the runs where the harness stayed healthy. SSH drops and disk-full errors get published as their own rates.
Reality
A dev.to proposal sorts a repo's tests into human oracles, seeded property checks and quarantined flakes, then requires all three classes to agree before a merge. The classifier it ships checks paths and expiry dates.
Reality
A dev.to field report spends 48 hours on one flaky test and ends at environment drift. What it hands over is four written constraints and a grid of locale, timezone, worker and file-descriptor settings.
Reality
Eleven passes and one failure on a footer change turned out to predate the diff. The test built tomorrow in UTC while the function counts days in America/Los_Angeles, so it goes red from midnight UTC to about 07:00.
Reality
The gate reads three committed ledgers that an agent job may read and not write: ten property seeds, a fixture hash file, and a flake freeze list capped at three. Its author publishes it as an unexecuted proposal.
Reality
Published flake numbers from Google, Microsoft, Atlassian and Slack describe an incentive system rather than a discipline failure, and the only lever that changes the arithmetic is a quarantine lane with a named owner.
Reality
A developer building a pairs-trading backtest found two of his own plans disagreeing on how many times to shift a series. His fix was a test that fires a price spike into the future.
Reality