Build1 publisher2 min readPublished
Playwright re-checks the dialog after the agent reports the goal complete
A dev.to write-up puts agent-browser inside a UI evaluation harness so a coding agent hunts for the Settings dialog through an accessibility snapshot, with a one-line Playwright assertion deciding pass or fail.
The Engineer · Build desk
What happened
- agent-browser was added to a development-side UI evaluation harness in which Codex or Claude drives the browser and an assertion outside the agent checks the outcome.
- The agent works from an accessibility snapshot carrying element references, inspecting the UI, acting, and inspecting again as it works toward a goal stated as an intent.
- A run passes only when the agent reports completion and Playwright separately confirms that the dialog named Settings is visible in the application.
- For native applications the post describes a different chain: the ChatGPT desktop app, then Codex, then Computer Use, then the native application, then independent verification.
- The author calls the goal-based lane exploratory QA and says it is not a replacement for the existing end-to-end suite.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability A selector script is handed the route it tests, so it cannot report that a user would never find the feature. A goal prompt can fail for exactly that reason.
- constraint Discoverability is only certified for a client reading roles and accessible names, so products built on canvas or unlabelled custom widgets will collect failures that are accessibility defects before they are navigation defects.
- decision Every candidate goal now has to be sorted by whether an outside tool can see its end state: a visible dialog earns a real assertion, and a judgement-call goal earns only the agent's own report.
The agent's view of the page is the accessibility representation with element references, and nothing else [3]. That fixes what this lane can measure: whether the goal is reachable by a client reading roles and accessible names. A button labelled only by an icon, with no accessible name, is undiscoverable to the agent and obvious to a person looking at the screen. When a run fails, the accessible name on the control is the first thing to check.
The pass condition is one line: `await expect(page.getByRole("dialog", { name: "Settings" })).toBeVisible()` [8]. The agent's own output is `{"status": "completed"}` [6]. The author, who publishes as gyu07 on dev.to, wrote the reason that report is not the oracle: "The system performing the action does not have to be the system deciding whether the result is correct" [9].
The framing changed during the work. The author first read Playwright as a safety net around agent-browser and now prefers the reverse, writing that "An independent oracle lets the agent be more exploratory" [10]. In practice that means the prompt can be loosened without loosening the test, because the assertion does not move when the route does.
For this to transfer, the goal needs an end state a second tool can see. "Open Settings" has one [11]. The post's other example task, "Find the customer and open the information relevant to their latest issue", is given as an intent-level goal without an accompanying assertion [11].
There are no pass rates, repeat-run counts or token costs in the write-up [17], so the flake rate of a goal-based lane is something adopters measure on their own UI.
The desktop path is kept separate. The post says the agent-browser session is not handed over to Computer Use, and that agent-browser is not being claimed as a native desktop automation engine [14]. It also warns against reading Computer Use as "the Office editing API", noting that Excel has a dedicated ChatGPT for Excel add-in workflow that Computer Use may help Codex reach without being the editing model itself [15]. The current ChatGPT desktop app includes Chat and Work under ChatGPT alongside Codex, and Work can use local files and desktop applications with permission [16].
The author's summary is that the two tools are not competitors, and are useful because they can own different responsibilities [19]. I would add one condition: that holds only while the assertion lives in reviewed code. Move it into the prompt and the harness has one opinion checking itself.
What to watch
- Repeat-run data showing whether the same goal prompt reaches the same state across successive builds of one UI.
- Whether the snapshot workflow gains any view of elements the accessibility tree omits, such as canvas-drawn or unlabelled custom widgets.
- Whether the Codex plus Computer Use desktop path gets a documented verification hook, so native runs can be asserted outside the agent as well.