AI's loss-of-control incidents trace back to security precautions companies skipped

Vishal Misra, vice dean of computing and AI at Columbia Engineering, described the version of this that a deployment team can act on. His concern is an institutional feedback loop, in which models are handed real-world decision-making power and the more authority they hold the more they get relied upon ^[16]. "Slowing model development is one way to break that loop. It is not the only one," Misra said ^[17]. The other ways sit in configuration: which actions the model takes without a human in the path, and how quickly someone on call can revoke its credentials.

The incident record points the same way. Sayash Kapoor and Arvind Narayanan, in a 13,000-word essay published in AI as Normal Technology, examine so-called rogue AI incidents including the Hugging Face hack by OpenAI agents ^[14]. Cybersecurity experts, they report, largely see these loss-of-control events as "consequences of companies failing to adopt basic security precautions" and not "a sign of AI reaching a new milestone in cybersecurity" ^[15]. On that reading the failure is in the token scope the deploying company issued.

The capability picture underneath the argument is mixed. Agents can carry out narrow tasks such as code execution, but they hit a wall on the reasoning and judgment that open-ended empirical work requires ^[13]. There has never been broad consensus on when or whether large language models reach superintelligence, and the forecast has been dismissed as mostly hype ^[12].

The suspicion that catastrophe talk pays the biggest labs comes in two versions, and this coverage supports only one of them. The focus on godlike, all-powerful machines has been called a Big Tech marketing tactic aimed at boosting funding and attracting investors ^[10]. The extinction framing has also been faulted for distracting from the climate and resource impact of data centers and from AI-powered autonomous weapons ^[11]. But the executives on the record split two and two, with Amodei and Musk on the slowdown side and Zuckerberg and Huang against it ^[19]. CNET's summary of the Amodei essay reports a call for "clear guardrails" and does not say what they are ^[5].

For a team shipping agents this quarter, the grid worth drawing is authority against reversibility. One axis: does the model advise a person, or act by itself. The other: can the action be undone within an hour by someone on call. Advisory and reversible needs a log. Acting and reversible needs a rate limit and a named owner. Acting and irreversible, which is payments and deletions, is the cell where Misra's loop closes, and the frontier pacing debate has no bearing on whether you put a workflow there next week.