Build1 publisher2 min readPublished
Thinkingbox grades 507 agent workflows on the backend state they leave behind
Every task in the benchmark is checked against its required terminal state, so any tool order can pass and any extra side effect fails. Claude Opus 5, the strongest model tested, cleared 66.50% at pass@1.
The Engineer · Build desk
What happened
- Thinkingbox-bench, described in a paper on arXiv, holds 507 policy-conditioned workflows across retail, travel and hospitality, auto insurance, neobank internal IT and consulting IT/HR support.
- Every attempt is graded by a task-specific executable check on terminal backend state, which accepts any valid trajectory while rejecting wrong, missing or extra persistent effects.
- Thirty of the 507 tasks add binary rubrics on the final response, covering required disclosures, confidentiality and consistency with the outcomes the agent actually executed.
- Claude Opus 5 was the strongest model tested across proprietary and open-weight systems, passing 66.50% of the tasks on the first attempt.
- The sandbox exposes domain tools through isolated MCP-compatible sessions and keeps complete execution traces, and both it and the benchmark are released at github.com/microsoft/thinkingbox.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Producing a score like this needs an expected terminal state written per task and a backend you can read side effects out of; a harness that stores tool calls and reply text cannot compute it, however many traces it keeps.
- exposure Failures that end cleanly and write valid state changes pass any monitor built on completion status or tool-call errors, so the wrong records reach whoever reads them next.
- decision Teams weighing an LLM judge against executable checks now have a published suite where 477 of 507 verdicts are reached without reading a word the agent wrote.
Scoring happens after the conversation ends. The sandbox runs the agent against a simulated user, retrieves the side effects from the backend session, then runs the task's own executable check on the terminal state [2].
The rejection rule is the part that costs work to implement. The checks accept valid trajectories and reject wrong, missing, or extra effects [3]. Under that rule, an attempt that issues the requested refund and then, helpfully, cancels the order fails. Writing such a check means holding an expected terminal state for every task, in domains chosen to stress multi-step transactions, policy-conditioned updates, record lookup and irreversible side effects [16].
Claude Opus 5 led the field at 66.50% pass@1, across both proprietary and open-weight models [6]. The abstract pairs that with 47.53%, and in the posted version the sentence breaks off before it names the second metric [7]. The drop is 18.97 points [8]. The paper's own gloss on it is the title, "One Success Isn't Reliability" [11], and the abstract's claim of "a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks" [10].
The finding I would take into a production review is about the cheap signals. The authors write that "many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion" [9].
For the 66.50% to say anything about a given deployment, several things would have to hold. The workflow has to end in state you can snapshot and diff, because that is what the check reads [1]. Each policy has to be expressible as an executable check. The user simulator has to behave like your users, since the score includes how the agent handles simulated follow-up [2]. And the workload has to be one where the persistent effect is the deliverable: only 30 of the 507 tasks score the final response at all, and the other 477 turn entirely on backend state [5][13].
The paper places itself alongside prior work that already checks final database states with policy-guided users (Yao et al., 2024), stateful conversational tools (Lu et al., 2025), and collateral changes in app worlds (Trivedi et al., 2024) [14]. Its claim is the substrate: one reusable loop supporting inference, leaderboard evaluation and failure analysis [15].
What to watch
- Whether the full paper defines the 47.53% metric and the k it uses, which sets where the reliability floor sits under the 66.50% headline.
- Per-domain scores: whether auto insurance and neobank internal IT track the overall pass@1 or drag it down.
- Whether the released repo ships the executable state checks in a form teams can point at their own MCP servers.