Published · 5h agoProduct2 min read
The 23.8-point harness spread is not in the record. What the record has is 5,194 runs
Harness-Bench sizes its study at 106 tasks and 5,194 trajectories, then calls the harness effect only "substantial". Meanwhile Anthropic deleted more than 80% of its own scaffolding.
Not a builder's beat, but builders have a standing stake in it.See today for builders
What happened
- Harness-Bench contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability and integrity.
- Across 5,194 execution trajectories, the Harness-Bench authors observe substantial variation in completion, process quality, efficiency and failure behaviour across model-harness pairings.
- The Harness-Bench authors argue agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone.
- The Harness-Bench abstract and introduction quantify the study's scale but describe harness effects only qualitatively, as substantial variation, without giving a numeric score spread.
- Harness-Bench code and data are available at github.com/Qihoo360/harness-bench, with additional resources on the project website.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
The figure a harness-only spread implies is not in the material we have. Harness-Bench is the one paper here that varies the execution layer on purpose, and it sizes its study precisely before declining to put a single number on the effect: 106 sandboxed offline tasks, 5,194 execution trajectories, and the phrase "substantial variation" covering completion, process quality, efficiency and failure behaviour [1][2][4]. Divide runs by tasks and you get exactly 49 trajectories per task [14]. That says the grid of model-harness pairings is wide enough to produce a large spread. It does not say what the spread was. The code and data are published [5], so someone can compute it; the abstract will not hand it over.
What the number would turn on is whose layer is being measured. Anthropic says it removed over 80% of Claude Code's system prompt for models like Claude Opus 5 and Claude Fable 5 with no measurable loss on its coding evaluations [7], having found it was overconstraining the agent through system prompt, CLAUDE.md and skills, to the point where a single request carried conflicting instructions such as "leave documentation as appropriate" against "DO NOT add comments" [8]. Read arithmetically, the deleted majority of that prompt was contributing nothing the coding evals could detect [15].
Those two results are not in conflict, and that is the useful part. Harness-Bench's variation spans efficiency and failure behaviour, and it names execution-alignment failures where plausible reasoning comes unstuck from tool feedback, workspace state and output contracts [2][6]. A coding eval that scores final artifacts sees none of that. So the scaffolding you delete on the lab's advice may cost you nothing on the lab's scoreboard and something on yours.
The absorption is not confined to prompts. Docker Desktop 4.86 swapped libkrun for Docker's own hypervisor on Mac, preserving the user's setting and switching on restart [11], and Go's proposal review group accepted a standard-library uuid package because google/uuid was already a staple import [13]. Absorption also arrives with fresh limits: Docker VMM has no Rosetta support, so amd64 emulation is slow, and MongoDB and Cassandra may fail on virtiofs [12].
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Harness-Bench contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability and integrity.
ReportedView cited source - [2]
Across 5,194 execution trajectories, the Harness-Bench authors observe substantial variation in completion, process quality, efficiency and failure behaviour across model-harness pairings.
ReportedView cited source - [3]
The Harness-Bench authors argue agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone.
ReportedView cited source - [4]
The Harness-Bench abstract and introduction quantify the study's scale but describe harness effects only qualitatively, as substantial variation, without giving a numeric score spread.
ReportedView cited source - [5]
Harness-Bench code and data are available at github.com/Qihoo360/harness-bench, with additional resources on the project website.
ReportedView cited source - [6]
Harness-Bench identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence or verifiable output contracts.
ReportedView cited source
Sources & coverage · 4 publishers
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- docs.docker.com5h agoVirtual Machine Manager | Docker Docs
