Yale physicist says fixing flawed problems raised AI physics scores to as high as 94%
Yale's John Sous says correcting errors in physics benchmark problems lifted cutting-edge AI scores from as low as 30% to as high as 94%. Well-defined problems may be within reach, but the case that physics will follow maths is still open.
The Scientist · Science desk
What happened
- Those low scores came from six separate tests built to assess AI on hard physics problems.
- Sous says the scores did not square with the successes he and his colleagues have had using the models.
- New Scientist reports the fault lay in tests that did not provide the necessary information, not in the AI.
- Formalisation has begun to be used to test physics, but New Scientist says it is early days compared with mathematics.
- In Logan Wright's Yale lab, Claude controls laser pulses and gathers data on how their shapes affect a rubidium atom.
Why it matters
- What it limits constraint A low score on a test with broken problems measures the test, so rankings on uncorrected physics benchmarks are a weak guide to which model to use for physics work.
- What doesn't add up contradiction The framing of physics as next in line for maths-style disruption sits against the account's own detail: formal checking is young in physics and Sous calls the full theorist's job unclear.
- Who has to decide decision A lab weighing AI for physics has a design question first: can its experiment be reduced to computer-controlled inputs and outputs, which Wright calls the key.
The method is to audit the problems, correct them and score the models again. If scores rise after correction, as New Scientist says they did [4], flawed items were costing the models points, a real finding about the instrument. Leaderboards rarely get this kind of check.
The two figures are range ends: models scored "as low as" 30% [2], and corrected scores reached "as high as" 94% [4]. Sous, who says models may have an unfairly bad reputation in physics because of how they are tested [1], told New Scientist: "I think anything that's well defined, even if it's very hard, I suspect that AI will be very good at it" [13]. He added: "This is backed up by our data" [14]. New Scientist does not say whether the 30% and 94% come from the same model and test, how many problems were changed, whether anyone corrected them before seeing model answers, or what the data Sous cites contain.
We think the first claim is plausible and the second is not yet earned. The comparison with mathematics rests on formalisation, which turns a mathematical idea into code that can be checked, so an AI knows when it has reached a correct solution [6]. The physics versions are young, and physics problems are often messier and harder to pin down [7]. On whether AI can do the full job of a theoretical physicist, defining, creating, exploring and testing, Sous said: "The picture is not clear yet" [15].
The nearer-term form of the idea is in Wright's lab. Wright says the key is to build experiments where inputs and outputs can be computer controlled, then hand that control to an AI [8]. He says Claude can predict what output a given laser pulse will produce and then test that prediction [12]. The apparatus does the checking that formal proof does in maths. Wright says such runs can go overnight and finish as much work as a human would in a week [10].
Wright is open about the limit. Not all of the universe can be explored this way, he said: "It'll be like very constrained domains" [11].
What to watch
- Publication of the corrected problem sets with a list of what changed, and whether corrections were made without sight of model answers.
- An independent re-run of the corrected benchmarks on several models, reporting scores per model and per test rather than a range.
- Wider use of formal checking for physics claims, which would give AI the same way of knowing it is right that maths has.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence40
- Adoption15
- Hype gap+20
- Incentives
- Not scored
- Confidence45
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
John Sous at Yale University says AI models may have an unfairly bad reputation when it comes to physics because of the way they are tested.
- [2]
Cutting-edge models have previously scored as low as 30 per cent on six different tests designed specifically to assess AI's ability on tough physics problems.
- [3]
Sous says the low scores do not square with the successes he and his colleagues have seen when using the models.
- [4]
Checking the problems in the benchmark revealed a litany of errors and ambiguous definitions; once corrected, scores rose to as high as 94 per cent.
- [5]
New Scientist reports that the problem lay not with AI but with the tests not providing the necessary information.
- [6]
Formalisation turns a mathematical idea into computer code that can be methodically checked, and it allows AI to process a problem and know when it has arrived at a correct solution.
- [7]
Some advances have been made in using formalisation to test physics, but it is early days compared with mathematics, and physics problems can often be messier and harder to pin down.
- [8]
Logan Wright at Yale says the key is to create experiments where the inputs and outputs can be computer controlled, so that control can be handed to an AI that pulls the levers to explore the problem space.
- [9]
An experiment in Wright's laboratory analyses how different shapes of laser pulses affect a rubidium atom, with researchers using Anthropic's Claude to control the laser pulses and gather data on their effects.
- [10]
Wright says such experiments can run overnight and complete as much work as a human would in a week.
- [11]
Wright said: "It's not going to be [the case that] all of the universe can be explored in that way. It'll be like very constrained domains."
- [12]
Wright says Claude can now say that it thinks a given laser pulse will produce a given output, and it can test that.
- [13]
Sous said: "I think anything that's well defined, even if it's very hard, I suspect that AI will be very good at it".
- [14]
Sous said: "This is backed up by our data."
- [15]
Sous said: "The picture is not clear yet." He was asked whether AI can do the full job of a theoretical physicist, with end-to-end research: define, create, explore, test.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- newscientist.comWill physics be the next field to be turned upside down by AI?
1 article · October 15, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI for ScienceFollow
- AI BenchmarksFollow
- Physics researchFollow
Entities
- John SousFollow
- ClaudeFollow
- New ScientistFollow
- Logan WrightFollow
- Yale UniversityFollow
- AnthropicFollow