Anthropic co-founder says a third-party-checkable kill switch may have to be mandatory

The question for whoever is shipping on a hosted model is narrow: what does "pull the plug" actually turn off. Clark said most labs, Anthropic included, have some way of doing it, and that policymakers may need to enforce having one ^[2]. His own framing was two questions. "Should you mandate for companies to definitely have a killswitch? Is that kill switch verifiable by a third party?" he asked ^[3]. "I think that's the kind of thing society is going to want to know and might want to eventually pass rules around," he said ^[4]. Clark told the BBC that the specifics of requirements and verification should be part of "the larger policy conversation" ^[5].

Verification is the harder half. Each lab built its own way to pull the plug ^[2], so an outside auditor would have to learn each one, and a team running two model providers for failover cannot assume the two switches cut at the same layer. A vendor can prove it is able to stop serving a model worldwide and leave its customer guessing whether the shutdown is scoped to the model or to one tenant's traffic.

The bill in Congress would also give certain government agencies the power to demand that a tool be turned off or limited ^[7]. The order goes to the model provider. Only the provider is party to it, and the outage lands on its customers.

The extinction percentages are the part of this that travels furthest and helps least with planning. Anthropic scientist Evan Hubinger said he personally thought the possibility of human extinction from AI was ">10% within the next decade" ^[13]. Geoffrey Hinton told the BBC on Friday that a 10% chance of AI killing all humans was "not unreasonable" ^[14]. Clark, asked for his own figure, said "I don't think these statistics are that useful", and said letting AI continue as a totally unregulated industry was a bad idea ^[15]. "We are rolling dice with immense risks," Clark said. "And the point is, we have to change the course of this industry" ^[16]. Of the three, two gave a number and one declined ^[21].

Dario Amodei called over the weekend for the pace of AI development to slow and be more closely monitored, and said any action to rein it in should come "without sacrificing commercial advantage" ^[10]. He did not say how development would be slowed ^[11]. Anthropic is reportedly preparing a potentially record-setting stock market debut ^[12]. Sam Altman said on Friday that OpenAI, valued most recently at $852bn, would not do the same this year because of the current debate around AI safety ^[19].

For a renewal conversation, two axes are enough. The first is scope: does the shutdown reach the whole service, or your deployment. The second is evidence: has anyone outside the vendor tested it. Scoped and externally tested is a control you can write a runbook against. Global and untested is a single point of failure that the vendor asserts works and nobody outside has checked, and the two mixed quadrants tell you which of the two questions to press first. Both Anthropic and OpenAI have self-reported incidents where AI agents acted in ways that were unexpected ^[17]. A mandated switch is for events like those. On the customer's side it arrives as an outage.