Product1 publisher3 min readPublished
Splitting a data room across sub-agents beats a standard tool-loop harness on Harvey's benchmark
Harvey and Baseten report rubric pass rates rising from 23.3% to 62.4% across seven models once the data room sits in a Python REPL and sub-agents do the reading, while Claude Code with Opus-5 manages 24.6%.
The Product Desk · Product desk

What happened
- Harvey and Baseten built a recursive language model harness that loads a whole M&A data room into a Python REPL, with a root agent delegating bounded review to sub-agents that each work in their own context window.
- Across seven models, mean rubric pass rate on Harvey's LAB Diligence tasks rose from 23.3% under the Legal Agent Bench tool loop to 62.4% under the new harness, a gain of 39.1 percentage points.
- A single LAB Diligence data room holds up to 5,000 documents across dozens of category folders and as much as 80 million tokens, more than one model context can hold.
- Reinforcement-learning post-training of a Qwen3.5-122B-A10B orchestrator inside the harness took pass rate from 29.9% to 63.0% on 50 held-out data rooms, according to Harvey.
- Two general-purpose coding agents were tested with web tools disabled and the same minimal instruction, and Claude Code with Opus-5 passed 24.6% of criteria.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- decision The choice in front of a platform team is no longer only which model to buy: Harvey's own comparison shows a frontier model inside a general coding agent barely clearing the baseline harness, which puts the budget argument on the dispatch layer instead.
- cost A 62.4% pass rate still leaves roughly 215 unmet items on an Aravon-sized rubric, so the review labour moves from reading documents to checking a memo against a checklist, and the associate pays for it.
- capability A post-trained mid-size open-weight orchestrator matching the seven-model harness mean makes a self-hosted root model a credible substitute for a frontier one, which is the part a firm with its own inference bill can act on.
- constraint Every number here comes from synthetic environments graded by an LLM judge on a benchmark its authors built, so it constrains what the result can tell you about a real data room with scanned pages and inconsistent folder names.
The most useful line in Harvey's write-up is about the failure, not the fix: in the baseline runs, agents searched and read selectively and left much of the data room unexamined [10]. That is a coverage problem wearing the costume of a reasoning problem. Harvey notes that some findings need several documents combined, and others need the agent to notice that supporting evidence is missing [21]. A model that opens a few dozen files out of five thousand [5] cannot see an absence.
The fix is a dispatch layer rather than a smarter reader. The root agent gets the corpus as queryable variables in a Python REPL, scopes the review, and fans sub-tasks out in parallel to sub-agents that each work in a bounded context and hand findings back as variables [1][13]. Harvey points out that deal teams in law firms cut the work the same way, by category [14].
The comparison that carries the argument is the coding agents. Claude Code with Opus-5, web tools off, given the same minimal instruction as the tool loop, landed 1.3 points above the tool-loop mean [16]. A capable agentic product plus a frontier model, without the corpus-in-a-REPL decomposition, sits roughly where the baseline sits. That is one data point, measured by Harvey inside Harvey's own benchmark, and it is still the cleanest evidence in the post that the architecture is doing the work.
Post-training tells a cost story more than a ceiling story. The reinforcement-learning run on the Qwen3.5-122B-A10B orchestrator bought 33.1 points [17], which lands 0.6 points above the seven-model harness mean [18]. A post-trained mid-size open-weight root does the orchestration job about as well as the average frontier root in the same harness. If you are hosting your own inference, that is the finding to price.
Now the arithmetic an operator has to live with. At the 62.4% mean, 37.6% of criteria fail; applied to the 571 criteria in Harvey's Aravon Bridge Bank rubric, that is about 215 unmet items in a memo the harness considers finished [15]. The grader is an LLM judge against an expert rubric [6], and the environments are synthetic [11]. Nobody in this experiment has fought a scanned indemnity clause or a folder named "misc final v3".
So this result is for the platform team choosing between a longer context window and building a decomposition layer, not for the associate, whose Friday still involves reading the memo against the checklist. What decides it is whether one task exceeds what a single context can hold, which Harvey says of every LAB Diligence room [22], and whether the review can be cut into slices whose merge a human can check, the way categories cut cleanly for a deal team. Both yes, and the points are in dispatch. If the work cannot be partitioned, none of these numbers transfer.
The share of the corpus an agent actually opens per task is the number that should settle the question before a bigger model gets bought. If it is reading three percent, a longer context window buys a more confident memo about the same three percent.
What to watch
- Whether the GLM-5.3 scale-up run Harvey says is under way as the root model beats the post-trained 122B orchestrator, and by how much.
- Whether the per-model spread behind the seven-model mean gets published, so a buyer can see what the model it already pays for gains.
- Whether anyone outside Harvey and Baseten reproduces the harness gain on non-synthetic data rooms with a human grader.