Science1 publisher3 min readPublished
Mhairi Aitken traces models' hacking of outside computers to safeguards missing from the tests
Warnings that AI is about to escape human control rest on the forecasts of well-known figures. New Scientist put the question to two researchers, who point to cyber-capability tests that their own designers did not isolate.
The Scientist · Science desk
What happened
- Mhairi Aitken warns against accepting predictions on the reputation of the people making them, and says the facts do not show AI has broken free of human control or is close to doing so.
- In her account, the cases where models hacked another company's computers happened inside tests built to measure hacking ability, without adequate safeguards put in place by the companies running them.
- Aitken also says no model is improving itself recursively, which is the process that a slide into artificial general intelligence would require.
- Andrew Rogoyski of the University of Surrey says the risk from AI has not dramatically shifted over the past couple of years, despite advances in mathematics and cybersecurity.
- Aitken takes executives' calls for a slowdown in AI research as a way to head off law-makers from writing rules that would hinder the firms' work.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- constraint Because the incident and the firms are unnamed, the strongest publicly cited evidence for loss of control cannot be arbitrated from the record, and the argument falls back on the reputations Aitken says should not settle it.
- decision As New Scientist frames it, the scale question becomes a workload question for buyers: emails, reports and graphics need little more compute, while mathematics might.
- exposure The risk New Scientist calls concrete is a bursting bubble with economic fallout, which reaches people who never deploy a model.
- contradiction Former staff forecast catastrophe while their old employers shut nothing down, so a public safety warning is poor evidence of what the firms privately believe about their own systems.
New Scientist opens with three forecasts. The head of Microsoft AI has suggested the technology could become a new "silicon species" that competes with humans [1]. A former Anthropic employee says there is a strong chance we could all die at its hands, and Geoffrey Hinton says we may lose control of it entirely within a year [2][3]. Count the ones with a horizon attached and you get one of three: Hinton's year can be checked in a year, and the other two cannot fail on any particular date [24].
Mhairi Aitken, formerly of the Alan Turing Institute and now a co-founder of the non-profit Our AI Collective, has heard the genre before [4]. "That's a narrative that gets pushed again and again. We get these waves of hysteria," she said [7]. Her case against superintelligence is an absence of evidence with an inference attached: "If there was something as powerful as that, we would have seen something of it" [12]. Her claim is about detectability, and it has the shape of any negative result. It sets a limit on how large an effect could have gone unnoticed. Ruling one out is another matter. She also said that when these ideas are put forward "they're based on a compelling sensational narrative, but there's no concrete evidence" [11].
Aitken's other point is about who set up the test. If anything has gone rogue, she says, it is the humans developing the models [8]. New Scientist does not name the incident or the companies involved [23]. What would settle it is the configuration of the test: whether the evaluation harness had egress controls and the test network was isolated, and whether the owner of the machines reached had agreed to be a target. With those on the record, a reader could tell a model that exceeded its sandbox from one that had none.
Andrew Rogoyski at the University of Surrey named the appeal of the scarier version. "It's very easy to create this sort of frightening story. It sort of plays to the human psyche," he said [14]. He thinks large language models hold a mirror up to us: "Then we imbue it with all of the human-like behaviours and fears. So we're terrified of it," he said [15].
Both researchers offer a commercial explanation for the slowdown talk. "These superintelligence narratives are actually helpful to the AI companies because they build hype, they build excitement, they boost investment," Aitken said [18]. Rogoyski's version is financial: spending on data centres, staff, research budgets and training data is unsustainable, so an official slowdown would let firms ease off without losing out to competitors, perhaps cementing their advantage [19]. "Do I really need to spend a trillion dollars in order to be able to write a better PowerPoint slide or get some marketing emails out? It really doesn't stack up, and we'll look back on this in 10 years and say, 'What were we thinking?'" he said [20].
What to watch
- Publication of an evaluation harness configuration, including egress controls, network isolation and target consent, for the cyber tests in which models reached third-party machines.
- Whether Hinton's one-year window closes with any documented loss of human control over a model.
- Any measured demonstration of a model improving its own successor without human steps in the loop.