Product1 publisher3 min readPublished
Four AI insiders put the odds of human extinction between 5 and 80 percent
Evan Hubinger, Dario Amodei, Emmett Shear and Daniel Kokotajlo have each published a personal probability that AI kills everyone. The figures span 75 percentage points, and Gizmodo reports that none of them rests on a formula.
The Product Desk · Product desk
What happened
- Anthropic alignment research lead Evan Hubinger wrote on X last Tuesday that the likelihood AI could kill all humans within the next ten years is greater than 10 percent.
- Anthropic cofounder and chief executive Dario Amodei has given his own p(doom) as a window running from 10 percent to 25 percent.
- Emmett Shear, who briefly replaced Sam Altman at the head of OpenAI in late 2023, told the Huffington Post that year that he oscillates between 5 percent and 50 percent.
- Daniel Kokotajlo, who left OpenAI in mid-2024 over concerns about its pace of development, gives a fixed value instead of a window and puts his p(doom) at 70 percent.
- Gizmodo reports that some skeptics read the warnings as marketing timed to the OpenAI and Anthropic public offerings, which are expected to value both companies in the trillions of dollars.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- constraint Without a defined event and a stated method, a probability cannot be updated. A team that copies one into a risk register will never see an observation that changes the cell.
- contradiction Newport treats the warnings as manipulation and Thomas says the fear is genuine, and neither reading helps the operator who has to explain the figure to a board, because both produce a number that cannot be checked.
- decision Anyone drafting an AI risk policy now has to choose between sourcing a likelihood from these lab figures and leaving the likelihood column empty while logging named beliefs.
- exposure Newport puts the labs' communication choices, not only their models, in the frame by arguing the harm to public mental health has outweighed what AI has delivered so far.
Somewhere a risk owner will be asked this quarter to put a likelihood on AI catastrophe into an internal register. The tidiest number on offer is one a lab researcher posted to a social feed last week. For a number to earn that cell, the event has to be defined tightly enough that two people would score it the same way, and the horizon has to be stated. Something observable also has to be able to move it.
Hubinger's two figures answer two questions. The greater-than-10-percent one is bounded at ten years [1]. The roughly 80 percent he gave earlier is for his "chance of existential risk from AI", with no horizon attached [6]. Hubinger has published nothing that converts one figure into the other, so the gap between them is not a forecast being revised [15].
The windows have the same problem in another form. Shear's is 45 percentage points wide and Amodei's is 15, which makes one exactly three times the other [13]. That ratio would tell you something about relative confidence if either window came out of a model. Gizmodo reports that nobody, including researchers at the frontier labs, has a real mathematical formula for these odds [7]. Taken together, the published figures cover 75 points, from a 5 percent floor to 80 percent [14].
The motive question is live, and both sides are on the record. Cal Newport, the author and Georgetown computer scientist, wrote in a New York Times op-ed earlier this year: "This could have been a period of hopeful innovation, but instead our emotions are being manipulated by Silicon Valley's self-serving and morally untenable addiction to doom trolling" [9]. Anthropic safety researcher Drake Thomas answered that charge in an X post last Wednesday. "I promise you, we are actually just fucking scared," he wrote, "it's not galaxy brained marketing" [11].
For the person filling in the register, the motive barely matters. In a spreadsheet, a sincere hunch and a strategic one behave identically: neither can be checked, and both invite multiplication by an impact score to produce a figure nobody can defend in an audit. Gizmodo's piece covers the statements and the criticism of them, and does not report any company or regulator writing a p(doom) into a risk framework [17].
The test to apply to any probability arriving from outside the building has two parts. First, what event, stated so that two colleagues scoring the same year would agree on whether it happened. Second, what observation between now and next year would move the number by ten points. Answer both and you have an estimate. Answer neither and what you have is a belief, and a belief belongs in the register as an attributed statement with a name on it. The cost of that discipline is real: an attributed belief cannot be multiplied by an impact score, so the likelihood column stays empty and the committee that wanted a colour-coded heat map goes without. Sam Altman, whose company sits in the middle of this, has said he has "never known how to put an exact number on p(doom)" [8].
What to watch
- Whether Hubinger or Anthropic publishes the reasoning or model behind the greater-than-10-percent figure.
- Whether the OpenAI and Anthropic IPO filings carry existential risk language, and in what terms.
- Whether any regulator asks labs to define the event and the horizon behind a published probability.