Gemini 4's safety report says bugs block any pass on about a quarter of audited ML R&D challenges
Google DeepMind's Gemini 4 safety report says bugs block any pass on about a quarter of audited ML R&D challenges. DeepMind estimates fixing them would not change its risk call, but a post on lesswrong.com asks whether the benchmark can still rule out the automation threshold.
The Engineer · Build desk
Three pass rates. About 75%: the maximum possible pass rate if a quarter of all 74 tasks cannot be passed. 90%: DeepMind's safety margin. 100%: the theoretical requirement for the ML R&D automation CCL.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Max pass rate if a quarter of 74 tasks can't pass | 75% about | 14 | |
| DeepMind's safety margin | 90% pass rate | 4 | |
| Theoretical requirement | 100% pass rate | 4 |
What happened
- The benchmark, GRB, holds 74 tasks that agents attempt in a sandboxed coding scaffold with a 2-hour timeout per task.
- The rule is a pass rate: a model would need 100% to reach the critical level, and DeepMind set a 90% margin to cover evaluation environment bugs.
- The audit also found several more challenges graded against details the task never stated.
- According to the post, some evaluations ran on checkpoints from 10 and 6 days before the final model.
Why it matters
- What it limits constraint GRB is internal, so the audit sentence is the only public evidence of task quality, and outsiders cannot rerun the repaired tasks.
- Who's exposed exposure The report promises that a ruled-out model is highly unlikely to have reached a critical level. A 25% defect rate tests that promise directly.
- Who has to decide decision Anyone citing Argon as below the ML R&D thresholds must choose between taking DeepMind's estimate on trust and waiting for a re-score on repaired tasks.
The audit shows the measure is damaged. It does not show the ruling is wrong, and the material does not say which way a repaired benchmark would move the score.
The report's examples of the damage are crashing verifiers and incorrect ground truth [1]. Those are harness failures. A verifier that crashes is hard to beat, and every bugged task becomes a fixed deduction from the pass rate.
DeepMind planned for some of this. Its 90% line, set below the theoretical 100%, exists to cover environment bugs [4]. That margin absorbs 10 points of lost tasks [16]. The audit found about a quarter, or roughly 25 points [1]. If that quarter spans the full 74-task suite [3], the best possible score is about 75% [14], which is 15 points under the 90% line [15].
Two caveats cut against that reading. The audit covered "the subset of original challenges that remain in the current suite," so the quarter may not span all 74 tasks [1]. And the post adds that the report's p.66 text says the 90% rule applies to the unbugged tasks [13]. If the threshold is computed on passable tasks, the ceiling is not 75%. The post quotes neither Gemini 4's GRB pass rate nor the working behind DeepMind's estimate.
That estimate is the report's answer. It states that "even if the model could in fact succeed at these affected evaluations, it would still not change our overall risk determination" [5].
The benchmark bears on one threshold more than the other. The automation level covers a model that "Can fully automate the work of any team of researchers at Google focused on improving AI capabilities, with approximately comparable all-inclusive costs" [7]. The acceleration level asks whether the model "has been used" to speed up AI development, and the post says the report checks that with evidence from inside Google [8]. A broken task suite weakens the first check directly. The second rests on other evidence.
Sergey Brin, quoted by Hugh Langley in Business Insider on October 15, said Google is "starting to get onto that RSI treadmill" [10]. He also said: "And now our researchers and the model work hand in hand to make the next version of the model" [11]. Neither line says a threshold has been crossed.
In our view, putting the audit in the report was the right call, and a margin for bugs is the right design. Whether 10 points is enough depends on how many of the 74 tasks the quarter covers.
What to watch
- A re-scored GRB result on repaired tasks, or Gemini 4's published pass rate, would show how close the model sits to the 90% line.
- A clarification of whether the 90% rule applies to all 74 tasks or only the unbugged ones, and whether the quarter covers the whole suite.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap+15
- Incentives
- Not scored
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The Gemini 4 Argon safety report (p.67) says that, in an audit of the subset of original challenges that remain in the current suite, about a quarter contained bugs that prevent any submission from passing (e.g. crashing verifiers, incorrect ground truth).
- [2]
According to the report's audit, several more challenges were graded against details not stated in the task.
- [3]
GRB (Google RE-Bench) currently contains 74 tasks spanning the pre-training and post-training lifecycle, and evaluates agents running in a coding scaffold within sandboxed environments, with a 2-hour timeout per task.
- [4]
For the ML R&D automation evaluations, a model theoretically needs to complete 100% of the benchmark challenges to reach the CCL, but DeepMind set its safety margin at 90% to account for evaluation environment bugs that might artificially prevent success. The decision rule is a pass rate.
- [5]
The report says: "our estimate is that even if the model could in fact succeed at these affected evaluations, it would still not change our overall risk determination." It also says the issues produce potential false negatives that risk underestimating the model's true capabilities.
- [6]
The report says it intentionally keeps its assumptions conservative so it is highly unlikely a model actually reaches a CCL if DeepMind has ruled it out.
- [7]
The Frontier Safety Framework's ML R&D automation CCL is a model that "Can fully automate the work of any team of researchers at Google focused on improving AI capabilities, with approximately comparable all-inclusive costs."
- [8]
The ML R&D acceleration CCL concerns a model that "has been used" to accelerate AI development, meaning acceleration that has actually happened, which the post says the report checks using evidence from inside Google.
- [9]
Some of the report's evaluations ran on checkpoints from just 10 and 6 days before the final model (p.160).
- [10]
Sergey Brin, in a roundtable discussion about Argon quoted by Hugh Langley of Business Insider on Oct 15 2026, said Google is "starting to get onto that RSI treadmill."
- [11]
Sergey Brin also said: "And now our researchers and the model work hand in hand to make the next version of the model."
- [12]
GRB is internal only, so the post relies on what the report says about it.
- [13]
The post adds a caveat that the report's p.66 text says the 90% rule applies to the unbugged tasks.
- [14]
If a quarter of all 74 tasks cannot be passed, the maximum possible pass rate is about 75%.
- [15]
A ceiling of about 75% is about 15 percentage points below the 90% line.
- [16]
A 90% safety margin against a theoretical 100% leaves room for 10 points of lost tasks.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- lesswrong.com"About a quarter of them contained bugs": Gemini 4's ML R&D benchmark
1 article · October 16, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Entities
- LessWrongFollow
- Google DeepMindFollow
- Frontier Safety FrameworkFollow
- Sergey BrinFollow
- Gemini 4 ArgonFollow
- GRB (Google RE-Bench)Follow
- METR RE-BenchFollow