OpenAI and Ironclad use production contract workflows to train and test computer-use agents

Most of the public detail on this partnership comes from a dev.to breakdown, and its author hedges. The pipeline is presented as "the likely flow" and the snapshot code as a "plausible schema" ^[13][1]. The post does not include pass rates, task counts or the models tested. It is one engineer's reconstruction of what an eval built on the case study would need ^[2].

Ironclad's flows are worth testing on for two reasons. They hold state, and different users take different paths through them ^[4][6]. Fixing the UI state and scripting the approvals ^[11] removes much of both. Scripted approvers are easier to grade than lawyers. What remains is structure: the workflow graph, the role boundaries, the screens each role sees, and failure cases taken from production ^[5]. That is still more than a simulated browser task, which the post says is what most computer-use benchmarks are ^[8].

The sketched grader has a real flaw. Call validate_agent_run and it walks the expected steps by index. It returns False if the agent's trace runs short. It returns False at the first step that does not match. It checks the success criteria only after every step has matched ^[1]. An agent that completes the contract by a different valid route therefore fails ^[7]. The same post says human users take different routes ^[6]. A grader that scores the end state against the success criteria, and treats the step list as one reference path, avoids this. Because the schema is the author's guess, Ironclad's real grader may already do so.

Version handling is the strongest part. Ironclad ships product updates, and a UI change breaks the eval. The post says the infrastructure has to track product version alongside eval version, keep older snapshots backward-compatible, and flag test cases a UI change invalidates ^[12]. I think this is the right design. When a pass rate drops, a version-pinned eval can tell a worse agent apart from a moved button.

The post's broadest claim is about transfer: "If an agent can navigate this, it can navigate most SaaS tools" ^[9]. For that to hold, the target tool needs contract approval's shape, with role-scoped permissions, conditional branches and approvals that move between teams ^[3][4]. Software without that shape learns little from an Ironclad score. The only workflow that settles the question for a given team is its own. This pipeline shows the cost of building that test. The logger has to capture DOM snapshots, API request and response pairs, intent signals and success criteria at every step ^[14]. People then classify each failure, a step the post says is not automated ^[15]. The evidence for a wider move away from synthetic benchmarks is this one partnership, described secondhand ^[2][13].