Kimi K3, the best open-weight model, solved none of First Proof's 10 research problems
ProofCouncil solved nine of First Proof's 10 research-math problems in round three, and open-weight Kimi K3 solved none. Mathematicians who avoid frontier models over data-leakage fears have no open-weight option that scored on this exam.
The Scientist · Science desk
ProofCouncil, the best performer, solved 9 of the 10 research problems in a manner the judges deemed acceptable with minor revisions for publication in a peer-reviewed journal.
- ProofCouncil, best performer, claim 4 9 of 10 research problems 90% computed
ProofCouncil, best performer The rest of the 10
| Share | Part | Whole | Percent | Claim |
|---|---|---|---|---|
| ProofCouncil, best performer | 9 research problems | 10 research problems | 90% (computed) | 4 |
What happened
- Ten problems, solicited from experts across subdisciplines, were answered as written with no human input, and three graders per problem scored each answer in a mock peer review.
- ProofCouncil combines a GPT 6 Astra agent with other agents that check its work, and the Kimi K3 entry used a similar harness.
- The round one winner in February got just a few problems right, and the round two winner in June got six or seven.
- First Proof requires entrants to be openly available, so OpenAI's new model was excluded from the round.
Why it matters
- decision Anyone weighing work on open math models now has an outside test on which the leading one scored zero, and Williams urged the community to "invest a lot into improving them".
- contradiction The report's pay-to-play framing fits poorly with two details: the winning entry is a multi-agent harness, and OpenAI's strongest model is closed to outside mathematicians, so a subscription alone does not buy access to it.
- constraint The proof task may soon stop separating frontier models: Abouzaid said testing them on it "might not be that relevant in the future".
On ten problems, the spread between the top entry and the best open-weight one is 90 percentage points: 9 of 10 against 0 of 10 [4][7][19]. A single problem moves a score by 10 points [20], so a difference of a problem or two between entrants would mean little. A gap of nine problems is well outside that.
Mohammed Abouzaid, a Stanford mathematician on the First Proof team, said the gap between open-weight and frontier models is "very large in research mathematics" and "much larger than in other domains" [9]. Lauren Williams, a Harvard mathematician on the team, said of Kimi K3: "It really fell flat on its face." [8] Scientific American's report says proprietary subscription-based models from frontier labs far exceed the math skills of open-weight ones [15].
Both top entries are agent harnesses, so the exam compares scaffolded systems [2]. The team also tested GPT 6 Astra and Claude Opus 5.5 [18], and producers of other major models, including Google, did not participate [3]. The report does not include scores for Astra or Opus 5.5 on their own, the cross-domain comparison behind Abouzaid's remark, or any subscription price or run cost.
Some mathematicians use open models regardless. Williams said, "I know people who use open-source models exclusively because they're so paranoid about data leakage with the frontier models." [10] The team has had many requests to test open-weight models from mathematicians concerned about privacy [16]. Abouzaid said, "We have a lot of questions from mathematicians telling us, 'I would like to look for alternatives to using OpenAI's or Anthropic's models, but I don't know whether they would work,'" [11].
We think the capability gap is established for one kind of task: proving a well-defined statement at research level, in a single round of ten problems. The exam does not test how a model does when a problem is not handed over as written. First Proof asked each model to answer the problems as written, with no further human input [1].
The larger claims are harder to check. For the release the report calls OpenAI's "slop drop", the company did not disclose the thousands of problems its new model attempted, most of which it did not solve [13].
What to watch
- Whether Kimi K3 or another open-weight harness scores above zero in a later First Proof round.
- A later round that scores bare models and agent harnesses side by side would separate model strength from scaffolding.
- Whether OpenAI opens its new model to outside mathematicians, which would allow the direct comparison this round could not make.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap+20
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
First Proof's mathematicians solicited problems from experts in a broad range of subdisciplines and selected 10; each model under evaluation was asked to answer the problems as written with no further human input, and three graders per problem evaluated the answers in a mock peer review.
- [2]
ProofCouncil is an aggregate model that combines a GPT 6 Astra agent with other agents to check its work; the Kimi K3 entry used a similar harness.
- [3]
Producers of other major models, including Google, did not participate in the round.
- [4]
ProofCouncil, the best performer, solved nine out of the 10 research problems in a manner the judges deemed acceptable with 'minor revisions' for publication in a peer-reviewed journal.
- [5]
In the round one results released in February the top model got just a few problems correct, and in the round two results released in June the winner got six or seven.
- [6]
First Proof's rules require entrants to be openly available, so OpenAI's new model was excluded.
- [7]
Kimi K3, the best open-weight model tested and described as the leading open-weight model, solved exactly zero problems.
- [8]
Lauren Williams, a Harvard mathematician and First Proof team member, said of Kimi K3: "It really fell flat on its face."
- [9]
Mohammed Abouzaid, a Stanford mathematician and First Proof team member, said the gap between open-weight and frontier models is "very large in research mathematics" and "much larger than in other domains".
- [10]
Williams said: "I know people who use open-source models exclusively because they're so paranoid about data leakage with the frontier models."
- [11]
Abouzaid said: "We have a lot of questions from mathematicians telling us, 'I would like to look for alternatives to using OpenAI's or Anthropic's models, but I don't know whether they would work,'"
- [12]
For the main skill First Proof tests, proving well-defined statements, Abouzaid said "the models have become very good," "so good that testing the frontier models on this task might not be that relevant in the future."
- [13]
For OpenAI's 'slop drop', the company did not disclose the thousands of problems its new model attempted, most of which it did not solve.
- [14]
OpenAI's internal model remains completely inaccessible to outside mathematicians, preventing a direct comparison of its performance with others.
- [15]
The results suggest that proprietary subscription-based models from frontier labs far exceed the math skills of more transparent, accessible open-weight models; Scientific American headlines the report as showing AI-powered math to be a 'pay-to-play sport'.
- [16]
The First Proof team has received many requests to test open-weight models from mathematicians concerned about privacy after recent high-profile controversies.
- [17]
Williams said: "The open models are really, really far behind, so we would urge the community to invest a lot into improving them."
- [18]
First Proof also tested OpenAI's GPT 6 Astra and Anthropic's Claude Opus 5.5, among other entrants.
- [19]
The spread between ProofCouncil and Kimi K3 is 90 percentage points on a 10-problem exam.
- [20]
With 10 problems, one problem moves a score by 10 percentage points.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- scientificamerican.comNew 'First Proof' exam reveals how AI-powered math is a pay-to-play sport
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI Model EvaluationFollow
- Open-Weight Model PublishingFollow
- AI Mathematics BenchmarksFollow