Swapping the harness under a fixed model kept pass rate within 8 points on 50 Terminal-Bench Pro tasks

The paper's proxy for the cost gap is the no-action turn: a turn in which the agent neither edits a file nor issues a new command ^[7]. It still costs a model call and a slot in the context window. It still burns wall-clock seconds. The authors describe these turns as "a per-task wait tax, not just a token tax" ^[8]. A model-only score does not recover any of that. The paper's stated concern is the developer picking an agent for daily work, who cares about cost per resolved task and time to completion, and about how much oversight it takes to keep the agent productive ^[17].

The one hard multiplier in the introduction comes from someone else's run. Terminal-Bench 2.0 reports Claude Opus 4.5 at 52.1 percent under one harness and 57.8 percent under another, on 256.9M and 3.9M input tokens respectively ^[9]. Divide those and the expensive configuration consumed about 66 times the input tokens ^[10]. As the paper pairs the figures, it also scored 5.7 points lower ^[11]. For that ratio to mean anything about your stack you would need the same model, the same tasks, the same turn cap and the same context-management policy. It is also total input tokens for a benchmark run, not tokens per solved task, which is the metric the 50-task study measures ^[2].

A gap in tokens per solved task is the paper's headline claim, and the abstract published on arxiv.org gives that sentence without the multiplier in it ^[5]. The study released anonymized configs, raw trial logs, aggregated snapshots and analysis scripts ^[13].

Terminal-Bench Pro has 400 tasks across 8 domains with 28 tests per task ^[14]. The study used a stratified 50 of them ^[2]. At that size one solved task moves the score by two percentage points, so the reported 0-to-8-point band covers zero to four tasks ^[15]. The paper's 95 percent paired-task bootstrap intervals include zero for every pairwise gap except the largest ^[4]. I don't think that shows three harnesses are equally accurate. At n=50, accuracy differences that small are not resolvable, and cost and latency are what this design can see.

Leaderboards rank entries by model name, with the harness undisclosed, varied to maximise the score in each row, or treated as an unstated constant ^[16]. SWE-bench and its successors noted that scaffold choice affects scores, and treated it as a confound to control ^[18]. Doing the measurement in-house costs a full grid: the three-harness, two-model design ran 300 trials over 50 tasks ^[3], and adding one harness and one model takes that to 600 ^[19].