Build1 publisher3 min readPublished
Month two of an unreviewed AI test pipeline left 30 percent of generated Playwright tests flaky
A dev.to post-mortem swapped 40 percent of an end-to-end regression suite for LLM-written Playwright tests over six months. Its two published numbers, 85 percent passing first run and 30 percent flaky in month two, count different tests.
The Engineer · Build desk
What happened
- A team replaced 40 percent of its end-to-end regression suite with LLM-generated Playwright tests in TypeScript, written from natural-language scenarios described by product managers and developers.
- 85 percent of the simple happy-path tests passed their first execution in CI, with the generator picking semantic locators such as getByTestId and getByRole.
- By month two, 30 percent of the generated tests were flaky, with failures clustered around dynamic content and waits the model did not write.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The pass rate is bounded by the DOM the generator is handed: it can only choose getByTestId where a test id exists, and teams without test ids or accessible roles get the CSS-selector output that went flaky.
- decision Teams copying this workflow have to decide what the merge gate actually checks, because a syntax-only pre-commit hook admits an assertion nobody awaits.
- cost The authoring saving is one-off, since boilerplate time falls to zero once per test, while flake triage recurs on every CI run until someone refactors the test.
- contradiction The post credits the generator with using Playwright's auto-waiting correctly and then prints a failure where the assertion is never awaited, so which prompts produced which output is left open.
The post-mortem prints two Playwright snippets, and they differ in more than complexity. The passing example locates a button with `getByTestId('new-project-btn')` and closes with `await expect(page.getByText('Project created successfully')).toBeVisible()` [9]. The failing example clicks `#save-button` and then calls `expect(page.locator('.success-message')).toBeVisible()` with no `await` [13]. A web-first assertion retries only while something is waiting on it, so in the second snippet the check fires once, against a page that has not finished its round trip [23]. The failing snippet also reaches for CSS classes, which is the locator style the post credits the generator for avoiding [10].
Where that lands depends on the merge gate. Generated code went through a local pre-commit hook for basic syntax checking and then into the main CI pipeline [5]. The post said no manual code review of the test logic was performed for the first two months, simulating a maximum automation scenario [6]. A missing `await` is valid TypeScript, so the hook parses the file and passes it [24].
The authoring win the post reports is real and narrow: correct module imports, proper `async/await`, no hallucinated Playwright methods, strictly typed output, and boilerplate time reduced to zero [21][20].
Month two carries the one failure figure, 30 percent of the generated tests flaky [12]. The generated tests were 40 percent of the suite [1]. Multiply the two and you get 0.12, so about one test in eight across the whole regression suite was flaking from this source alone, assuming the 30 percent is measured on the replacement set [22]. The 85 percent is not the complement of that. It counts simple happy-path tests passing on first execution in CI, while the 30 percent counts generated tests two months in [25].
The post defines the quantity it set out to measure as "the percentage of tests that remained stable and green without manual refactoring over six months" [16]. It reports that the rate was initially low and then improved dramatically after strict prompt engineering guardrails and post-generation linting, without giving a value [17][18]. The numbers that do appear are the 85 percent first-run pass rate and the month-two flake rate [7][12].
For the 85 percent to transfer, your DOM has to look like theirs. The generator could use `getByTestId` because the button already had a test id [9]. Where elements have no test ids and no accessible roles, the same model writes the CSS-selector version, which is the one that went flaky. The comparison also runs against a low baseline: the suite being replaced was a legacy Cypress suite the post describes as slow, flaky and difficult to read [4]. Failure modes, per the post, followed the limits of LLM context windows and the model's lack of understanding of application architecture, including network latency, server-side rendering delays and database transaction locks [19][15].
What to watch
- A published survival-rate figure with its denominator would show whether the prompt guardrails changed outcomes or only the intake of tests.
- Which lint rules the team added post-generation, and whether they are type-aware enough to catch an unawaited assertion.
- Whether the flake rate moved after month two on the same set of generated tests, once test-logic review started.