Codex agent solves 49 of 50 unpublished physics problems for under $3 each
GPT-6-Astra, running in OpenAI's Codex, solved 49 of 50 unpublished theoretical-physics problems on the TheoPhysEval benchmark. The problems are routine research calculations, and the authors say AI still lacks the "taste" for breakthroughs.
The Engineer · Build desk
Ranking of AI agents by percent of 50 TheoPhysEval problems solved: GPT-6-Astra in Codex 98, Opus 5.5 in Claude Code 92, Gemini 3.8 Flash 68, open-weight DeepSeek-V4.1-Flash 54, open-weight GLM 5.3 42.
| Rank | Item | Value | Claim |
|---|---|---|---|
| 1 | GPT-6-Astra in Codex | 98% | 3 |
| 2 | Opus 5.5 in Claude Code | 92% | 10 |
| 3 | Gemini 3.8 Flash | 68% | 10 |
| 4 | DeepSeek-V4.1-Flash, open-weight | 54% | 10 |
| 5 | GLM 5.3, open-weight | 42% | 10 |
What happened
- Anthropic's Opus 5.5 in Claude Code came second at 92 percent, ahead of Gemini 3.8 Flash at 68, DeepSeek-V4.1-Flash at 54 and GLM 5.3 at 42.
- With internet access removed, Astra and Opus each scored 92 percent, so web search made surprisingly little difference.
- The authors say a leading expert in the subfield needs about a day or less for a typical problem, and specialists from neighboring fields take much longer.
- Follow-ups with the experts found small, hard-to-spot errors in several reference solutions, and once those were fixed Astra's answer was correct every time.
Why it matters
- capability Problems of the black hole kind needed a handful of qualified experts and several days each before modern agents; a commercially available agent now does them with no human hints.
- constraint The score applies to problems that pair a known method with a slightly modified question and a verifiable answer. Choosing which question to ask is outside the test.
- contradiction Agents score higher on TheoPhysEval than on CritPt although the authors rate their own problems harder. The article offers grading errors in other benchmarks as one explanation, so cross-benchmark rankings need the answer keys checked first.
- decision Whether a calculation agent gets internet access is a choice with a measurable price here: three problems out of 50 for Astra.
The black hole problem shows what the agent has to do. It finds an older paper, carries its equations from five-dimensional to four-dimensional spacetime, and then writes its own program to compute the answer [7]. The coordinator, Suvrat Raju, who co-wrote the older paper, took about a day [8]. The 35 authors, from ICTS, Harvard, KU Leuven, Google DeepMind and elsewhere, often built problems by tweaking a few values in a calculation from a recent paper of their own, so the new result could not be guessed from the old one [1][6].
On that problem, Gemini and GLM copied the older paper's formulas unchanged in the web search run, and those formulas apply to five dimensions, not the required four [11].
The cost figures are close but not identical. The source's headline says $2.40 a problem [23]. Its text gives an average of under $3 and just under $116 for the whole benchmark [22]. Divided across 50 problems, $116 is $2.32 [24]. The article does not say which runs or what accounting sits behind each figure, so we would plan on "under $3" and treat the cents as unsettled.
The authors read the offline result as evidence that the top language models have, in effect, committed much of the theoretical physics literature to memory, since many problems depend on formulas from papers the models usually did not receive [25]. If that reading is right, part of the 49 of 50 is a measure of what sits in the weights. In our view, a calculation whose method was never written up would score lower.
With web search, the one problem Astra missed also stumped every other model. That problem required finding an equation that generates a specified curve [13]. The experts answered it with special mathematical forms that qualify as the required differential equation only in a broader sense [14]. The authors admit the problem might seem "unfair" but consider it realistic, because researchers often have to infer the underlying physics from imprecise data [15].
The benchmark leaves out major open questions by design [5]. It cannot test taste. The researchers say a missing sense of "taste" is one reason breakthroughs haven't followed yet [18].
What to watch
- Outside reruns of the Hugging Face dataset and test setup that reproduce 49 of 50 with the corrected answer keys.
- Any clarification of which runs the $2.40 headline figure and the under-$3 average each cover.
- A benchmark of open-ended problems where picking the question matters, the part TheoPhysEval leaves out.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap+15
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Thirty-five researchers from institutions including the International Centre for Theoretical Sciences (ICTS), Harvard University, KU Leuven, and Google DeepMind released TheoPhysEval, a benchmark of 50 unpublished research problems in theoretical physics with verifiable answers.
- [2]
According to the authors, a leading expert in the relevant subfield would need about a day or less to solve a typical problem without AI help, and specialists from neighboring fields would take much longer.
- [3]
GPT-6-Astra, running in OpenAI's Codex coding environment, found the right answer for 49 of 50 problems, or 98 percent.
- [4]
The team tested commercially available AI agents and gave them no human hints.
- [5]
The collection deliberately leaves out major open questions and focuses on what the authors call 'turn-the-crank' problems, where researchers apply known methods to a slightly modified question; problems like these are often handed to PhD students as a first step into research.
- [6]
Contributors often tweaked a few values in a calculation from one of their own recent papers so the new result couldn't be guessed from the old one.
- [7]
The black hole example requires finding an older paper, carrying its equations over from a five-dimensional to a four-dimensional spacetime, and computing the result with a custom program.
- [8]
Coordinator Suvrat Raju, a co-author of the older paper, needed about a day to solve the black hole problem.
- [9]
The authors estimate that before modern agents arrived, only a handful of experts had both the physics knowledge and the experience with these computational methods, and even they would have needed several days.
- [10]
Anthropic's Opus 5.5 in Claude Code came second at 92 percent; Google's Gemini 3.8 Flash scored 68 percent; the open-weight DeepSeek-V4.1-Flash and GLM 5.3 reached 54 and 42 percent.
- [11]
On the black hole problem, Gemini and GLM copied the formulas from the older paper unchanged during the web search run; those formulas apply to five dimensions, not the required four.
- [12]
Without web search, Astra and Opus each scored 92 percent; internet access made surprisingly little difference.
- [13]
With web search, Astra solved everything except one problem that stumped every model; it asked for an equation that produces a given curve.
- [14]
The experts' solution to that problem relies on special mathematical forms that only count as the required differential equation in a broader sense.
- [15]
The authors admit the problem might seem 'unfair' but consider it realistic because researchers often have to infer the underlying physics from imprecise data.
- [16]
For several problems, follow-ups with the experts revealed small, hard-to-spot errors in their reference solutions; once those were fixed, Astra's answer was correct every time.
- [17]
The agents do better on TheoPhysEval than on some other physics benchmarks, such as CritPt, even though the authors consider their own problems harder on average; a recent study found some theoretical physics benchmarks are graded incorrectly and in some cases badly underestimate what the models can do.
- [18]
The researchers behind the benchmark say a missing sense of 'taste' is one reason breakthroughs haven't followed yet.
- [19]
The problems and test setup can be downloaded as a dataset on Hugging Face.
- [20]
A 92 percent score on 50 problems is 46 problems solved.
- [21]
Astra solved three fewer problems offline (46) than with web search (49).
- [22]
GPT-6-Astra's average cost was under $3 per problem and just under $116 for the whole benchmark.
- [23]
The source's headline states the cost as $2.40 apiece.
- [24]
Just under $116 across 50 problems works out to about $2.32 per problem.
- [25]
The authors take the small web-search effect as a sign that leading language models have effectively memorized large parts of the theoretical physics literature; many problems depend on formulas from research papers that the models usually didn't receive.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
2 independent publishers whose own reporting we read for this story.
- marginalrevolution.comAstra Held Its Ground (Truth)
1 article · October 15, 2026
- the-decoder.comGPT-6-Astra solves 49 of 50 physics research problems at $2.40 apiece
1 article · October 15, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.