The graders sit outside the agent's own view of the system. Task graders, rubrics, private seeds and mock implementations stay outside the tree the agent can see [5]. The agent gets a business request, a workspace and access to local software replicas, and every run begins in a fresh container [4]. When it stops, a host-side verifier reads the final state of the mock services and any files produced, and the task passes only if every required check passes [6]. The familiar failure mode, where a model narrates a plausible sequence of actions and reports success while the database or storefront is untouched, scores zero here [7]. Lian and Xie also contribute to Business Arena, which extends the same state-based thesis to a simulated cross-border shop over a longer operating period [26].
For the scores to mean anything on your stack, your stack has to resemble the 14 offline replicas the suite ships, covering product publishing, freight booking, storefront administration and payment operations [8]. Then there is the mix: 53 of the 107 tasks are command line, 28 browser, 16 file and document, 10 API or MCP [3]. Command-line work is 49.5% of the suite [25], so an aggregate number is mostly a statement about shell competence. If your agents drive a browser session against a live storefront, that aggregate is answering a question you did not ask.
Now the ranking. Qwen3.8-Max's 156 passes across 321 harness-task combinations work out to 48.6% [10]; DeepSeek V4 Pro's 154 come to 48.0% [12]. The gap sits entirely inside one harness: ties in Pi and OpenClaw, then two tasks ahead in Accio [13]. That is six tenths of a percentage point [23]. The spread inside Qwen's own row is nine tasks between Accio and OpenClaw [18], four and a half times the margin that decides the open-weight ordering [24]. Claude Opus 5, which moved six tasks between its weakest and strongest harness [19], finished 35 passes ahead of Qwen regardless [15], at 59.5% and 10.9 points clear [14].
The code is Apache 2.0 and the task data CC BY 4.0, with commercial use permitted [9], so nothing stops a rerun. What runtimewire reports as missing is an immutable public bundle at task level [22], and its read of the tables is that they come from Alibaba's own benchmark and should be treated as a published result rather than an independent audit [21]. The practical effect is that nobody outside the team can name the two Accio tasks carrying the strongest-overall claim. Checking them means standing up 321 runs of your own, for a result that separates the harnesses by six tenths of a percentage point.
One integration detail is worth flagging while hunting for the source of harness spread. The repository's Qwen documentation notes the model returns its reasoning trace in a separate response field [20]. A harness that reads that field and a harness that ignores it are not running the same agent.
The ceiling is the figure I would take to a review. The best result anywhere in the tables still left 41 of 107 tasks unfinished [16], and the source's own conclusion is that a 61.7% pass rate over listings, freight, payments and customer records demands supervision and transaction-level checks [17].
