Pasting the bug report and page object file cuts maybe 60 percent of a Playwright test's typing, author says

The prompt behind that one-minute failing test has two parts. The first is the bug report, which the QA team writes in prose: the cart total does not update when you remove the last item while a coupon is applied ^[4]. The second is the page object file, which names the methods a test is allowed to call ^[5]. With both in context the model has enough to emit the imports, the fixture setup and the navigation steps, and the author puts that at maybe 60 percent of the typing ^[6].

Neither file says what the cart total should read after the removal, and an assertion needs that third input. The author's explanation is that the model writes for the happy path because the happy path is what is in the code, and that it will never ask what happens if the payment webhook arrives twice, because that question comes from being burned by a duplicate webhook at 2am ^[9]. Roughly 80 percent of the real bugs caught in that suite came from tests nobody would think to generate, according to the post ^[10]. "A generated assertion checks that something exists. A written assertion checks that something is correct," the author wrote ^[15].

The claim being tested here arrived in a PR channel, where a teammate pasted a generated Playwright file and wrote "AI generated this in 4 seconds, why are we still writing tests by hand" ^[1].

Two of the conditions for the 60 percent are inputs you must already own: bug reports written in plain language by people who reproduce them ^[4], and page objects that already cover the screens under test ^[5]. The third is training data for your stack. The author attributes the drop on mobile web to a mountain of desktop web test code in training. He reports that answers to questions about viewport behaviour, touch targets and a sticky header eating a click on a 390px screen come back confident and desktop ^[11]. On Flutter the output is plausible code calling an API that changed two versions ago. For that reason he built the smoke and regression structure by hand, with the model contributing boilerplate inside each test and nothing about the structure ^[12].

Step four of the published workflow is 20 local runs before the test goes near CI ^[14]. The flaky test used as the example in the same post fails once every 20 runs ^[7]. If failures at that rate were independent, 20 runs would catch at least one about 64 percent of the time, and a one-in-twenty flake would clear the gate in roughly a third of attempts ^[18].

On the feature every AI testing tool sells, the post is blunt: "a locator that silently repairs itself is a locator that stops telling you the UI changed. Sometimes the UI changing IS the bug" ^[13]. The argument is about what a failing test is for, and it holds whether or not model quality improves.

The post includes no measured data. Its figures are hedged in the original as "maybe 60 percent", "Roughly 80 percent" and right "maybe half the time" ^[19], from one engineer with three years in test automation and six months of heavy model use ^[3]. The closing line is the one worth arguing with: "AI made me faster at writing tests. It did not make me better at knowing which tests to write" ^[16].