Gemini 4 Argon falls from 97.5% to 49.8% on ARC-AGI-3 once vendor features come off
Google DeepMind's Gemini 4 Argon scored 97.5% on ARC-AGI-3 at a $4,800 run cost in ARC Prize testing, second to GPT-6 Astra's 99.9%. The score depends on vendor-specific harness features, so it says less about a team's own workload than the headline implies.
The Engineer · Build desk
Under the Standard Harness, API problems meant Argon finished 42 of 55 games at low reasoning, 46 of 55 at medium and 19 of 55 at high, where ARC Prize published no score.
- Low reasoning, finished, claim 6 42 of 55 games 76% computed
Low reasoning, finished The rest of the 55
- Medium reasoning, finished, claim 6 46 of 55 games 84% computed
Medium reasoning, finished The rest of the 55
- High reasoning, finished, claim 6 19 of 55 games 35% computed
High reasoning, finished The rest of the 55
| Share | Part | Whole | Percent | Claim |
|---|---|---|---|---|
| Low reasoning, finished | 42 games | 55 games | 76% (computed) | 6 |
| Medium reasoning, finished | 46 games | 55 games | 84% (computed) | 6 |
| High reasoning, finished | 19 games | 55 games | 35% (computed) | 6 |
What happened
- Under the Standard Harness, which allows only simple notes between rounds, Argon scored 49.8% at low reasoning for $5,800.
- API problems kept Argon from finishing all 55 games: 42 at low reasoning, 46 at medium and 19 at high, where ARC Prize published no score.
- Argon used 62% fewer actions than the human baseline on the 25 public games and matched or beat the speedrun record on three of them.
- On Epoch AI's Capabilities Index Argon scored 165, tying Fable 5.1 and landing just behind Opus 5.5 at 167 and GPT-6 Astra at 166.
- Epoch AI's early estimates put Argon eighth on its Math index and sixth on its Software Engineering index, though it leads several knowledge, agent and vision benchmarks.
Why it matters
- What it limits constraint The 97.5% holds only where retained reasoning and compaction are switched on. An agent loop that keeps its own notes sits closer to the standard-harness condition.
- Who has to decide decision ARC-AGI-3 scores play in unfamiliar turn-based games. A team buying Argon for the software engineering work Google pitches it for should weight Epoch's SWE index over this headline.
- What it costs cost The $4,800 run cost is priced at an introductory $2 per million input tokens and $10 per million output tokens. A budget built on that figure changes when the price does.
- What it sets up precedent ARC Prize has accepted general public API settings since the July dispute, so vendor-feature results are verified entries. Anyone comparing models has to check which harness produced each score.
Between the two harnesses, Argon's ARC-AGI-3 score moves by 47.7 points [18]. The difference comes from what the model is allowed to keep between turns. The Provider Adapter Harness preserves the model's opaque reasoning state and compacts long conversations automatically [3]. The Standard Harness is provider-neutral and lets a model carry forward only the notes it chooses to keep [5].
Astra shows the same drop. It fell from 99.9% to 62.7% under the standard harness [10]. The ordering survives, but the spacing does not. Argon trails Astra by 2.4 points with the vendor features on [21] and by 12.9 points with them off [22]. Thirteen of the 55 games went unfinished at the low-reasoning setting [19], and the published figure includes partial credit from those aborted runs [6].
The harness dispute dates to July 2026. GPT-5.6 Sol scored 7.8% in the standard environment. On OpenAI's Responses API with Retained Reasoning and Compaction it scored 38.3% and passed Claude Opus 5 at 30.2% [13]. According to The Decoder, OpenAI argued that benchmarks always measure the technical environment too [17]. ARC Prize co-founder Francois Chollet then allowed general, publicly available API settings [14].
ARC Prize has said that solving ARC-AGI-3 under the standard harness, without the extra scaffolding, is the more meaningful bar for progress toward AGI-level efficiency [15]. The low-reasoning standard run also cost $1,000 more than the adapter run [20].
For an internal evaluation, we would run the task set twice: once with state retention and compaction on, once with notes only. A wide gap between the runs means the vendor feature is part of what you are buying. A narrow gap means the harness matters less for your workload. Until that data exists, we think Epoch's domain indexes are the better prior for coding or math work.
What to watch
- Whether ARC Prize publishes a standard-harness score for Argon at high reasoning once the API issues that stopped 36 of 55 games are resolved.
- Whether Epoch AI's early Math and Software Engineering rankings for Argon move as more evaluations come in.
- Whether Google changes Argon's introductory token pricing, which every cost figure in the ARC Prize runs depends on.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence72
- Adoption
- Not scored
- Hype gap+35
- Incentives
- Not scored
- Confidence70
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Google DeepMind's Gemini 4 Argon scored 97.5% on ARC-AGI-3 at a cost of $4,800 using the Provider Adapter Harness, a setup that uses vendor-specific features like retained reasoning and context compaction.
- [2]
Only GPT-6 Astra scores higher on ARC-AGI-3, at 99.9 percent; according to ARC Prize, Google is the second lab after OpenAI to cross 95 percent.
- [3]
The provider adapter harness preserves a model's opaque reasoning state between turns and automatically compacts long conversations.
- [4]
In the Standard Harness, which only allows simple notes between rounds, Argon dropped to 49.8% at low reasoning, at a cost of $5,800.
- [5]
The standard harness is provider-neutral and lets a model carry forward only notes it chooses to keep.
- [6]
API problems prevented Argon from completing all 55 games in the Standard Harness: at low reasoning it finished 42 games (49.8%), at medium 46 games (46.3%), and at high only 19 games, so ARC Prize did not publish a score. Reported results include partial points from aborted runs.
- [7]
Costs were calculated using Gemini 4 Argon's introductory pricing of $2 per million input tokens and $10 per million output tokens.
- [8]
Google introduced Argon on September 30 as its first entrant in the Gemini 4 line, built for long, complex workflows spanning software engineering, enterprise knowledge work, and cybersecurity defense.
- [9]
Argon used 62% fewer actions than the human baseline across the 25 public games, and matched or beat the speedrun record on three of them and came within 10% on six others.
- [10]
When GPT-6 Astra was tested under the standard harness, its score fell from 99.9% under the provider adapter harness to 62.7%.
- [11]
Argon scored 165 on the Epoch Capabilities Index, tying Fable 5.1 and landing just behind Opus 5.5 (167) and GPT-6 Astra (166); the ECI combines over 50 benchmarks into a single capability scale.
- [12]
According to Epoch AI, Argon leads several knowledge, agent, and vision benchmarks, but early estimates put it eighth on the Math ECI and sixth on the Software Engineering Index (SWE ECI).
- [13]
In a July 2026 dispute, GPT-5.6 Sol managed only 7.8% in the standard environment; when tested with OpenAI's Responses API, including Retained Reasoning and Compaction, it jumped to 38.3% and beat Opus 5 at 30.2%.
- [14]
ARC Prize co-founder Francois Chollet then allowed general, publicly available API settings.
- [15]
ARC Prize has said a model that can solve ARC-AGI-3 under the standard harness, without the extra scaffolding, is the more meaningful bar for what the organization would consider progress toward AGI-level efficiency.
- [16]
ARC-AGI-3 drops an AI agent into unfamiliar, turn-based game environments with no instructions and measures how efficiently it figures out the rules.
- [17]
OpenAI argued that benchmarks always measure the technical environment too.
- [18]
Argon's ARC-AGI-3 score differs by 47.7 percentage points between the provider adapter harness and the standard harness at low reasoning.
- [19]
Thirteen of the 55 games went unfinished at Argon's low-reasoning standard-harness setting.
- [20]
The low-reasoning standard-harness run cost $1,000 more than the provider adapter run.
- [21]
Under the provider adapter harness, Argon trails GPT-6 Astra by 2.4 points.
- [22]
Under the standard harness, Argon's best completed configuration trails GPT-6 Astra by 12.9 points.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
2 independent publishers whose own reporting we read for this story.
- Gemini 4 Argon Scores 97.5% On ARC-AGI 3 With Harness, Only Second Behind GPT-6 Astra
officechai.com
1 article · October 15, 2026
- the-decoder.comGoogle's Gemini 4 Argon is closer to the AI frontier than any Google model in months
1 article · October 16, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI BenchmarksFollow
- Evaluation harnessesFollow
- Frontier AI modelsFollow