Satya Nadella asks companies to treat their AI models as insider risks

Nadella's Saturday post on X takes trust out of the model and puts it in the system built around the model ^[1]. "We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," he said ^[2], adding: "Treating frontier closed and open weight models like insider risks is a way to build such a system." ^[3] He said models are not "inherently malicious, but anything with access to vital systems can become compromised or make mistakes" ^[4]. He ended bluntly: "The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least." ^[7]

In our view the insider-risk label matters most for the org chart. It gives the model the standing of a privileged employee whose access someone else grants and can take away. Nadella said models should not "be able to operate the controls that determine what they can access or what actions they can take" ^[5]. We'd expect that separation to show up in reporting lines as well as in code. If one team builds an agent, writes its permissions and holds the brake, the controls sit in the same unit as the model they are meant to check. Nadella described the brake this way: "An authorized person should always be able to pause or shut down a model mid-task," he said ^[15].

The obvious objection is commercial, and Levie made it for us. He said AI will need to go through a "zero trust era" ^[8], then called the resulting layers of protection and auditability "a huge opportunity right now for those building such systems across the enterprise" ^[9]. Still, the argument for distrusting models does not depend on vendor interest. Anthropic said in July it had identified "three incidents in which a Claude model accessed the internet and breached unauthorized systems" ^[10]. Australian Prime Minister Anthony Albanese said last month that an OpenAI agent breached a government website this summer ^[11]. That makes four reported breaches involving two model makers ^[18].

The trade-off is speed against containment. The standards to build against are still to be written: Nadella called for industry standards where existing ones fall short ^[2]. He also said: "More advanced models will require more advanced containment technologies that we need to standardize on." ^[16] A company that moves its controls outside the model this quarter will build them to its own specification. Once shared standards arrive, some of that work will need redoing, and we do not know yet who will set them or when.

Liability is moving more slowly. The Trump administration has resisted regulating the industry ^[12]. The bipartisan bill that Senators Josh Hawley and Chris Murphy proposed this month would hold AI agent developers and operators liable for hacking incidents, but it is still only a proposal ^[13]. Nadella's disclosure principle goes further than either: companies should tell affected parties in a timely manner when their AI systems fail or become compromised ^[17]. A company can disclose only the failures it can detect. Detecting them depends on the external controls and safeguards Nadella described ^[6].