A green rerun is not a repair: self-healing tests need a merge gate outside the healer

A column published on devops.com argues that when a self-healing system inspects a changed page, proposes a new locator and reruns the test to green, that green result does not prove the test was repaired ^[1]. The consequence is a control problem rather than a tooling one: the author's recommendation is to treat an AI-generated repair like any other untrusted code change, with the healer allowed to propose the patch and a separate deployment gate deciding whether it is safe to merge ^[4].

The failure mode is specific. A replacement locator can resolve to the wrong button, a hidden duplicate, or an element from a different part of the page, and the run still passes because the automation found something clickable ^[2]. The piece calls this a false heal and rates it worse than an ordinary failure, because a red test creates work while a false heal makes the suite look healthy as its signal degrades ^[3]. In the worked example, a redesign breaks the selector for a checkout Submit button, the healer picks another button with similar text, and the test goes green even if that element is Cancel, a hidden mobile control, or a Submit button belonging to a different form ^[6].

Most self-healing demonstrations, according to the article, stop at one of two checkpoints: the replacement locator can be inserted, or the test runs without an error. Neither answers whether the repaired test still exercises the intended behavior ^[5]. The structural objection is that the healer cannot grade its own repair using the same evidence it used to produce it, so it needs an external contract describing what the test is supposed to touch and what outcome must follow ^[7]. If one agent proposes the change, relaxes the assertion and declares the rerun successful, it can make its own work easier to pass; a deterministic gate stops the repair process from editing the definition of success ^[12].

The proposed gate has three checks. Target identity records what made the original element the intended target, such as accessible role, name, a stable test identifier, the containing form or a nearby label, and requires the replacement to satisfy the same constraints rather than merely resolve to one element ^[8]; the piece notes Playwright's locator guidance favors user-facing attributes such as roles and labels because they carry more meaning than a long CSS path ^[9]. Behavior preservation reruns the user path and checks whether the expected request fired, the page reached the right state and the original assertion still ran, and fails any repair that bypasses or weakens the assertion even when the command exits cleanly ^[10]. Review scope puts the original locator, the proposed replacement, the matched element, the diff and the rerun evidence into one packet, with human approval required on release-critical paths, financial actions, account changes and security controls, and a lighter policy for lower-risk repairs where evidence is still retained ^[11].

The retention ask is the part most pipelines will fail today. Teams normally keep the final patch and the test result; the article wants six further artifacts kept as well ^[13][17], including every rejected candidate, without which the audit trail shows only the answer that passed and cannot say whether a confidence threshold was narrowly cleared or a reviewer overrode the recommendation ^[14]. That record is also what makes rollback tractable when the next UI change makes the repaired test behave differently ^[15].

The healer keeps a job in this design: collecting failure evidence, inspecting the page and ranking replacement locators as a candidate generator ^[16]. What to watch is whether vendors ship the gate rather than the ranking, and whether any of them expose rejected candidates and matched-element evidence as retained artifacts instead of transient logs ^[14][16].