CI cannot tell a regression from a stale test because nobody wrote the intent down

A developer writing on dev.to set out to make CI answer the one question every red end-to-end run raises and no report answers: did the app break, or did the test go stale ^[1]. That distinction is the whole triage decision, because one answer means stop the merge and fix a regression while the other means the product changed on purpose and the test is describing a version that no longer exists, and both arrive as the same red X ^[2].

What Playwright actually hands over is a line like `locator('[data-testid="place-order"]') timed out after 5000ms`, which the author calls a fact about a selector rather than a fact about the product ^[3]. The failure knows the selector, the timeout and the line number; it does not know what `place-order` is, that the button is the last step of a checkout flow, or that the pull request hid it behind a new express-checkout summary ^[5]. Closing that gap is manual work: open the trace, remember what the button does, find the PR, read the diff, decide ^[4]. The author's point is that none of this information is hard to obtain, it just never gets written down anywhere a machine can read it ^[6].

The conventional fix, wrapping actions in `test.step('add two items to the cart')`, produces a nicer report ^[7]. It fails as an input for two reasons: the string is hand-maintained, so it drifts and then confidently describes behaviour that no longer exists ^[8]; and it is prose, naming no control and carrying no selector, so there is nothing in it a program can match against a diff ^[9].

The change is to name the control instead of describing the step, declaring objects such as `new Button(page, 'place-order', 'Place order')` ^[10]. Each control carries three things: its kind, the selector it resolves to, and the human name ^[11]. The readable log falls out of that for free ^[12], but the artifact that matters is a per-action JSON record of action, control type, name, selector and status ^[13]. The run then leaves behind a record of which parts of the UI the test touched at runtime, which the author distinguishes sharply from which lines of code executed ^[14].

That gives two sets with a shared key, since a pull request is itself a set of changes to markup and test IDs ^[15]. Non-empty intersection means the PR changed the exact element the test was reaching for, so the test is stale; empty intersection makes it a regression candidate ^[16]. The rule is a heuristic on that key: a PR that edits the markup behind a selector and also genuinely breaks it lands in the stale bucket ^[2].

The anecdote worth keeping is the first attempt, which shipped the error and trace to an LLM ^[17]. Against a demo PR that deliberately hid the Place order button and removed a cart row, two tests failed ^[18], and the model returned several well-written paragraphs blaming an unawaited second add-to-cart in a shared `beforeEach` ^[19]. No such bug existed; the fixture was fine and the PR's own description said the row removal was intentional ^[20]. The author's reading is that the model reasoned well from insufficient input, since a broken fixture is a reasonable hypothesis given only the symptom ^[21]. Both failures should have been classified stale under the intersection rule ^[1].

Worth watching: whether this holds outside test-ID locators, and whether the evidence extends past one demo PR with two failing tests ^[18].