The guardrail lives in the call signature. In the new builder a step cannot exist without an assertion attached: no tap, wait, or type without declaring what the screen should show afterward, so a run fails at the step where reality diverged instead of four actions later when something downstream breaks [11]. Waiting is no longer a separate action an author decides to perform. Under the old API it was, and `pause(1000)` was the cheapest way to perform it, seeming to work locally and usually passing in CI [6].
Bypasses still exist. Custom timeouts and script injection are reachable, prefixed `UNSAFE_`, and Shopify treats `UNSAFE_timeoutInSeconds` in a test as a signal for review [13]. The bad path stays available but gets renamed into something a reviewer has to defend, and that renaming is the entire enforcement model. Shopify's account describes no retry policy in either direction [21]; what it documents is the removal of an affordance.
One layer down, the assertion changed meaning. Old tests often checked that a node existed in the component tree rather than that a merchant could see or use it [7]. New ones screenshot each step and locate the target visually, with PaddleOCR reading text and OpenCV matching against SVGs from Shopify's Polaris design system [15]. The second half of that stack is the part that does not travel. Icon matching needs a first-party design system whose assets are the same SVGs the app renders; text matching needs screen copy stable enough to recognise. Shopify has both.
The headline pair reads harder as failures than as successes. Half stability means one run in two fails; 98% means one in 50, so the per-run failure rate went from 50 to 2, a factor of 25 [18]. That factor sits on an undefined denominator. The post does not say over what window, across how many runs, or whether stability is counted per test or per suite run [19]. The 50% baseline is also a property of one test corpus written against one old API on Shopify's largest app [3], not a measurement of Appium.
What got cheaper is unambiguous. Adding a step used to mean opening an inspector, drilling into the component tree to find or add a `testID`, then wiring it up; now it is `touch({ text: 'Save' })` against what is visible in the simulator [17]. Shopify also argues the small surface area and predictable grammar make humans and AI tools produce correct tests on the first try more often [14], which is a claim about authoring, not about stability.
The mechanism that couples every action to its assertion works without OCR and without a design system; it simply needs an API that offers no way to skip the wait [11].
