With reasoning effort set to none inside OpenAI's Provider Adapter, Astra scores 96.7%, which is 34 points above the same weights running at maximum reasoning inside ARC Prize's standard harness [6]. The setting vendors sell as the capability lever lost to a configuration choice about state management.
That choice is small and legible. ARC Prize's standard harness gives every model the same minimal interface and lets the model decide which notes to carry forward, while OpenAI's adapter preserves the model's opaque reasoning state between requests and compacts longer conversations [5]. Whether the model keeps its own scratchpad, and who trims context when it fills, is the entire delta between the two numbers.
The cheaper run was also the better one. The standard harness cost $26,098 for 62.7% [2]; the adapter cost $18,817 for 99.9% [3]. That is $416 per point against $188 per point [2], and the losing run cost 38.7% more in absolute dollars [3]. On the 167 game-reasoning pairs both harnesses solved, the adapter used 49% fewer tokens and ran roughly 3.66 times faster [7]. Scaffolding shows up on the capability line and on the invoice at the same time.
Two limits apply to how far that finding travels. This is one benchmark family and one model, so nothing here shows the same harness dominance on retrieval or support triage. And the pinned figure is not perfectly pinned: Francois Chollet posted 66% where ARC Prize's table says 62.7%, a 3.3-point spread depending on which artifact you read, and the blog is the primary record [12][4]. ARC Prize says it will publish both harness results side by side from now on [11].
The rest of the launch numbers moved as well, and each metric was altered on its own terms. Fortune's Emily Forlini compared archived snapshots of OpenAI's launch post and found five metrics altered after publication [13]. Astra's hallucination rate read 4.2%, then 2% by 5.20pm, then 4.2% again [14]. Anthropic's Fable 5.1 fell from 87.8% to 78% on FrontierMath before settling at 83% [15], and Sol's ExploitBench doubled from 5.5% to 11.5% at a reasoning level Sol does not offer commercially, which OpenAI told Fortune it is investigating reverting [16]. The embargo draft put ARC-AGI-3 at 98.6% against 99.99% in the live post [17]. Not every edit favoured Astra, since two Anthropic HealthBench Professional scores went up [23]. OpenAI's account to Fortune is noise of a few percentage points from checkpoint, scaffold and evaluation run [18], and Snorkel AI's Vincent Sunn Chen told Fortune that scores routinely shift in the final hours before a launch [21]. Anka Reuel and Mike Hardy, the Stanford researchers who call this benchmaxxing, went to the system card for method and found barely any details about the internal hallucination evaluation, including no count of test items [19][20].
For an eval spec, the forcing function is four fields filled for both sides of every comparison: which harness ran it, whether reasoning state persists between requests, who compacts context and at what threshold, and what reasoning effort was set to [4]. Then draw the grid, same weights against different weights on one axis, same harness against different harness on the other. Only the same-harness cells say anything about a model. The different-harness cells price your platform team's work, which is worth knowing and is not a model claim. The comparison that travelled, 99.9% against Sol's 7.8%, sits in the fourth cell with both variables loose [8], and the publisher that later documented the gap had carried that pairing without the caveat in its own launch coverage [9].
