A Passing AI-Generated Test Is Not Evidence: Seven Checks Before It Enters Your Suite

A post on dev.to by aiwithanton sets out seven checks to run against an AI-generated test before you let it into the suite: name the user risk, break the product, check the final result, change the data, read the failure, repeat the run, then ask whether the test could stop a bad release ^[1][2]. It matters because the thing your assistant hands you is a file that passes, and the author's position is that a clean file, plausible names and a green run prove nothing about value ^[3].

The standard the piece sets is narrow: a useful test catches a failure that matters to users, and the review has to find that proof before the test is kept ^[4].

Check one is a sentence, not code. Name who loses something and what they lose; money, time, access and trust are clear answers, "the feature breaks" is not ^[5]. The worked risk is "A customer sees the wrong total and pays too much"; the rejected version is "checkout may fail", because it names no damage and gives the test no target ^[6].

Check two is the one that does the work. The article's sample Playwright test opens /checkout, clicks Pay, and asserts that the text "Success" is visible, which checks a message rather than the charged amount ^[7]. So you remove the payment action or return the wrong total. If the test still passes, according to the author it protects nothing useful ^[8]. Of the seven steps, this is the only one that requires you to induce a failure rather than inspect or rerun a passing test ^[18].

Check three moves from steps to results. The stronger version asserts that order-total reads $120.00 and payment-status reads Paid ^[9]. The post is precise about the division of labour: Playwright supplies assertions that wait for results, but choosing the right result is still yours ^[10].

Check four is data. AI tends to produce one neat example, while real users bring missing, wrong and extreme values, so the piece loops an amount field over zero, 999999 and the string "wrong" and requires an alert each time ^[11].

The last three checks are about cost after the fact. Run the test against a broken result and read the message: "Expected: $20.00 / Received: $120.00" points at the product error, while "Timeout after 30000ms" sends the next engineer into the logs ^[12]. Run the same input twice, which surfaces shared data, timing problems and hidden dependencies on other tests, and compare both runs rather than accepting a green second pass ^[13]. Then name the release the test would block, for example "This test blocks checkout when totals are wrong"; keep it if the answer is clear, rewrite or delete it if it stays vague ^[14].

That final gate is a queue-management decision as much as a quality one. The claim is that teams do not need every generated test, only the small set that proves important behaviour ^[15]. The author puts the whole review at a few minutes per test ^[17].

Two of the seven checks produce a sentence rather than a code change ^[16]. Those are the two most likely to be skipped under delivery pressure, and they are the ones that decide whether the other five are aimed at anything. Watch whether your team ever deletes a generated test after review. If nothing is ever discarded, the gate is decoration.