Ex-OpenAI researcher Cammarata argues AI neurons may be easier to interpret than long assumed
Independent researcher Nick Cammarata, who says he left OpenAI in 2023, published a report arguing almost the entire AI network is interpretable. Business Insider calls the findings still theoretical, so safety teams have a claim to test before they have an audit method.
The Board Room · Leadership desk
Builders see what goes in and out of AI models, not necessarily what is between. For years researchers assumed neurons packed unrelated things. Until recently researchers largely tracked chain-of-thought. Cammarata says networks are knowable. Business Insider calls it still theoretical.
- What it limits constraint AI model builders Know what goes into a model and what comes out, but not necessarily what happens in between, claim 2
- What it sets up precedent AI researchers For years assumed AI models packed unrelated things into a single neuron to save space, claim 3
- What it sets up precedent Frontier AI researchers Until recently, largely tried to understand how AI thinks by tracking chain-of-thought reasoning, claim 5
- What's now possible capability Nick Cammarata Says AI neural networks are knowable so long as you look carefully, claim 12
- What it limits constraint Business Insider Describes the findings as undeniably dense, couched in jargon, and still theoretical, claim 4
| Who | How | Kind | Claim |
|---|---|---|---|
| AI model builders | Know what goes into a model and what comes out, but not necessarily what happens in between | constraint | 2 |
| AI researchers | For years assumed AI models packed unrelated things into a single neuron to save space | precedent | 3 |
| Frontier AI researchers | Until recently, largely tried to understand how AI thinks by tracking chain-of-thought reasoning | precedent | 5 |
| Nick Cammarata | Says AI neural networks are knowable so long as you look carefully | capability | 12 |
| Business Insider | Describes the findings as undeniably dense, couched in jargon, and still theoretical | constraint | 4 |
What happened
- Until now the prevailing view was that AI neurons were uninterpretable, with no way to guess why a given neuron fired at, say, a picture of a cat, Business Insider reports.
- In Cammarata's example, one neuron might react to both a cat and the front of a car, perhaps because it registers the triangle shape the two share.
- OpenAI chief scientist Jakub Pachocki warned in a blog post last month that AI agents could obfuscate their visible reasoning and lie to human minders undetected, Business Insider reports.
Why it matters
- Who has to decide decision Staff time spent testing Cammarata's claim comes out of chain-of-thought monitoring, the method researchers have mostly relied on until recently.
- What doesn't add up contradiction The old assumption said neurons pack unrelated things together to save space. Cammarata's shared-feature reading says a neuron firing for a cat and a car is tracking one thing, and only that reading makes it auditable.
- What's now possible capability A reliable neuron-level reading would let builders inspect the step between input and output, where today they can see only what goes in and what comes out.
Safety teams audit a model largely through what it says about its own reasoning. Business Insider reports that frontier researchers have, until recently, mostly done this by tracking chain-of-thought reasoning [5]. Pachocki, OpenAI's chief scientist, said: "With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all" [7]. In our view Cammarata's report goes after this problem directly, because a reading of neurons does not depend on the model choosing to verbalize anything [12].
Cammarata published the report on Thursday, according to Business Insider [10]. He says he may have found sensible patterns in the way neurons respond to stimuli [12]. "Almost the entire network is interpretable if we look closely enough," he wrote [14]. Business Insider calls the findings dense, couched in jargon and still theoretical [4]. Its account does not say which models Cammarata examined or whether anyone has reproduced his results.
One researcher's hypothesis, relayed by one outlet, is not yet a method. Still, a claim that would give teams an inspection route independent of the model's own account is worth testing before anyone designs an audit around chain-of-thought alone.
Andrew Curran, whom Business Insider calls a prominent AI analyst, said on X: "I had long since given up on reaching any real level of interpretability before superintelligence arrived" [8]. He continued: "I stopped writing about it because of how demoralizing it was. Hard to overstate how big this may be" [9]. His reaction records how low expectations had fallen.
The horizons differ. A theoretical result changes no audit this week. Over a quarter, the practical step is replication: a team that reruns the patterns on its own model learns whether the claim holds there. If it does, next quarter's audit design can include a check that does not rely on the model's account of its reasoning. If it does not, the team has ruled out one route and keeps chain-of-thought monitoring as its main method.
What to watch
- A replication or rebuttal of the neuron patterns on a frontier model from an interpretability team at OpenAI, Anthropic or another lab.
- A fuller account of which models and tests Cammarata used, since Business Insider describes the report as dense and still theoretical.
- Any lab response to Pachocki's warning that proposes audits which do not depend on a model's verbalized reasoning.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence38
- Adoption
- Not scored
- Hype gap+35
- Incentives
- Not scored
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Previously the prevailing wisdom was that AI neurons were ultimately uninterpretable, meaning there was no way to guess why a specific neuron would fire when a model saw, say, a picture of a cat.
- [2]
Those building AI models know what goes into the AI and what comes out, but not necessarily what happens in between.
- [3]
For years AI researchers had assumed that AI models packed unrelated things into a single neuron to save space.
- [4]
Business Insider describes the findings as undeniably dense, couched in polysyllabic jargon, and still theoretical.
- [5]
Until recently, researchers at frontier AI companies have largely tried to understand how AI thinks by tracking an AI's self-reported thinking, called chain-of-thought reasoning.
- [6]
Jakub Pachocki, OpenAI's chief scientist, warned in a blog post last month that AI agents could now obfuscate their visible reasoning, leaving open the possibility that an agent could lie to its human minders and the minders wouldn't be able to tell.
- [7]
"With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all," Pachocki said.
- [8]
"I had long since given up on reaching any real level of interpretability before superintelligence arrived," Andrew Curran, a prominent AI analyst, said on X.
- [9]
Curran continued: "I stopped writing about it because of how demoralizing it was. Hard to overstate how big this may be."
- [10]
An independent researcher who formerly worked at OpenAI published a report on Thursday arguing that how AI models think may be easier to decipher than thought.
- [11]
Nick Cammarata said he left OpenAI in 2023.
- [12]
Cammarata says he may have discovered patterns in how neurons react to stimuli that make sense, and says AI neural networks are knowable so long as you look carefully.
- [13]
In one example Cammarata gives, a single AI neuron might react to both a cat and the front of a car, perhaps because the AI sees the triangle shape the two share.
- [14]
"Almost the entire network is interpretable if we look closely enough," Cammarata wrote.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
2 independent publishers whose own reporting we read for this story.
- businessinsider.comThis new theory on how AI ‘thinks’ could impact whether it kills us all
1 article · October 16, 2026
- implicator.aiGoodfire's Cammarata Says AI Neurons Are Readable After All
1 article · October 16, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.