Mitchell's complaint: we grade models with human exams when the control group is animals and infants

Melanie Mitchell, a cognitive scientist and computer scientist at the Santa Fe Institute, told Quanta Magazine's podcast The Joy of Why that the field lacks adequate methods for measuring machine cognition, and that AI is better described as a form of "alien intelligence" that operates through non-human cognitive mechanisms ^[2][3][4]. Her proposed remedy is not another leaderboard: adapt the methods psychologists already use on the two alien intelligences they cannot interview, babies and animals ^[5].

The stake is operational, not philosophical. Quanta frames the question as whether a language model answering a question is reasoning like a human or producing text that looks like reasoning ^[1], and notes that the answer determines what we can trust these systems to do, how closely they need supervising, and what their real-world impact turns out to be ^[7].

Follow that framing to its consequence, and a lot of current benchmark reporting stops meaning what it says. If two different mechanisms can produce the same score on a test, the test does not tell you which mechanism produced it ^[14]. That is the whole discipline of infant and animal cognition work: you cannot ask a crow to explain itself, so you design a control condition where the ability you are claiming is absent but every other route to the right answer is still open. Benchmarks built as human exams do not carry that control. They were written for subjects who share our failure modes, and they are scored as if a pass licenses a claim about internal process. Mitchell does not use the word unfalsifiable in the material we have; that inference is ours.

The episode's cautionary tale is a horse from the early 1900s that appeared to do arithmetic, offered as a warning about how we assess intelligence ^[8]. Quanta's introduction does not name the animal ^[15]. The point survives the omission: the performance was real and the attributed mechanism was wrong, which is exactly the error a score-only evaluation cannot catch.

Co-host Janna Levin put the epistemic problem plainly, saying we are excited about the artificial mind while having very little comprehension of the human mind, and that we are trying to skip a step ^[11]. Steven Strogatz pointed to comparative psychology, the study of intelligence in birds, dogs and dolphins, as the neglected reference class ^[12]. Mitchell had appeared on the show roughly five years earlier, before ChatGPT arrived around November 2022 ^[13]; the hosts date this recording to July 23, 2026 ^[10], which puts it about 44 months into the era of widely deployed language models ^[16].

Mitchell lays out six principles for better assessing machine cognition ^[6]. The Quanta introduction announces them without enumerating them, and the transcript portion available to us covers only the opening setup ^[15], so the substance of the proposal is in the audio rather than in anything quotable here.

What to watch: whether those six principles include control conditions of the kind animal work requires, and whether any lab reports them alongside a benchmark number. Watch also how interpretability is positioned, since the episode treats the difficulty of reading what happens inside these systems as an open problem rather than a solved check ^[8], and whether the AI-assisted mathematics results discussed in the same conversation ^[8] get evaluated as capability or as process.