Product1 publisher3 min readPublished
Rishub Jain quit DeepMind over who reads the code that builds the next model
WIRED reports that Jain left Google DeepMind in June and that Jacob Coxon resigned from Anthropic this week. Both named the same missing step, a human reading the work that builds the successor model.
The Product Desk · Product desk
What happened
- Rishub Jain left Google DeepMind in June, telling WIRED he was uneasy that he might not have proper visibility into how an AI model was building its successor.
- Researcher Jacob Coxon announced his resignation from Anthropic this week, warning that AI firms are racing straight to self-improving superintelligence and gambling with our lives.
- A senior Anthropic leader who works on AI safety, quoted by WIRED without a name, put the chance that AI kills all humans above 10 percent within the next decade.
- No frontier lab claims to have achieved a fully autonomous cycle of improvement, and WIRED reports it remains theoretical, even as it has drawn well-funded startups such as Recursive Intelligence.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- contradiction Both readings of the pipeline sit in the same WIRED report: no lab claims a fully autonomous improvement cycle, while Jain says the coding acceleration that begins one is already routine. The dispute is over where the loop starts counting, not over what is happening.
- constraint A safety process that only tests finished model outputs cannot answer this objection, because the artifact Jain says is missing is a record of who read the change that produced the successor.
- precedent Soares had been advising lab staff to quit and hearing that it would change nothing; Coxon's departure gives the next person weighing it a worked example to point at.
The step Jain objected to happens in ordinary engineering teams every week. Someone hands a coding task to a model, the model returns work nobody typed, and the reviewer either reads it or waves it through. According to WIRED, what changed for Jain was the thing being built: the next generation of models, with his own judgement progressively out of the path [1][2]. He told WIRED that "AI progress is increasing" and that "as AI becomes more capable, it poses more risks" [3].
Scale is what makes that hard to fix by adding reviewers. Daniel Kokotajlo, the author of AI 2027, told WIRED the current version of this work often means dispatching thousands of agents to collaborate on one problem, and said the complexity involved pushes oversight and control further away [12].
Nate Soares, a computer scientist at MIRA and coauthor of If Anybody Builds It, Everybody Dies, told WIRED that "I do think that the vision of recursive self-improvement is spooking people" and that "it's starting to feel real" [7]. On the technical field meant to keep models matched to human values, he said many people had entertained a fantasy that alignment would get easier as these things got smarter, that it is instead getting harder, and that they are reacting with alarm [8].
The most quotable line in the piece came from a senior Anthropic leader who works on AI safety and is not named: "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade" [6]. Nobody outside the company can check that number. The other claim can be checked from inside one, by asking whether a named person read the change that went into the successor model.
On the size of the exodus, the evidence is thin and worth saying so plainly. WIRED calls Jain "one of a growing number of AI researchers speaking out over those fears" and names two departures, his from Google DeepMind and Jacob Coxon's from Anthropic [21][18]. There is no attrition figure in the piece and no number for how much of a frontier model's code is now written by models. The dated sequence is firmer: Jain quit in June, and in July more than a thousand top AI engineers signed an open letter calling for a coordinated slowdown, roughly a month later [1][13][19]. Kokotajlo said the concern was growing well before Coxon's resignation, before the hacking incidents and before the math result [14].
For a team that lets agents write code that ships, the smaller version of Jain's problem has two axes: whether you can reproduce the change the agent made, and whether you can name the human who approved it. Both present is normal code review. A reproducible change with no named approver is where a lot of teams landed after turning on agent mode, and it is the box Jain describes one size up [2]. Neither present is the state teams tend to discover during an incident review. The rule that closes it, a named approver on every agent-authored change, costs merge speed, and that cost is the answer to give when someone asks why throughput fell.
Soares said he regularly talks to people inside the big AI labs who are worried about the consequences of their own research. "I tend to recommend they quit, and they say it wouldn't do anything," he said. "And then Jacob quits, and we see who was right" [11].
What to watch
- Whether any frontier lab publishes what share of its next model's code was model-written, and who reviewed it.
- Whether Anthropic responds publicly to Coxon's resignation, or names the safety leader who put the odds above 10 percent.
- Whether the July slowdown letter picks up more signatories from inside DeepMind, OpenAI and Anthropic.