Build1 publisher3 min readPublished
Microsoft's SRE Agent Autonomously Handles 'Safe' Mitigations After Humans Set Guidelines and Coached It
Azure SRE Agent correlates telemetry, blast radius and recent deploys before an engineer joins the bridge, and at some Microsoft teams more than half of incidents now close in auto mode because the action was pre-classed as safe.
The Engineer · Build desk
What happened
- More than 3,000 Microsoft service teams use the agent to investigate issues, perform root cause analysis, fix code, enable automatic mitigation and report at scale.
- Lead program manager Shamir Abdul Aziz says more than half of incidents at some internal teams are handled with no human intervention, limited to restarts, scale outs, rollbacks and escalated change orders.
- On InEight's first incident with the agent it identified the affected product, traced the root cause and recommended scaling Redis, while the DevOps team had been considering scaling the app service.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision The human sign-off moves from the incident to the list. Adoption work is enumerating which mitigations may execute unsupervised, and that enumeration happens weeks before the outage it governs.
- constraint The correlation only pays where telemetry, deployment history and change records are reachable from one place. A shop whose signals sit in several observability tools does the joining itself first.
- cost The cost is front-loaded onto the humans who write the guidelines and, in Abdul Aziz's word, coach the agent. They also own every action the agent takes inside those bounds.
- capability If Mehta is right that deterministic queries cannot catch a risky deploy in flight, then buying this is buying a probabilistic judgement about deploy risk, with the false positives that implies.
The boundary gets drawn before the pager goes off. The list Shamir Abdul Aziz, lead program manager for Azure SRE Agent, gave The New Stack is short: a restart, a scale out or a rollback of a service, plus change order requests escalated by customers [6]. A team can pre-authorise an action only if it can name the action in advance and either reverse it or cap it.
So there are two approval models inside one product. In the incident scenario The New Stack describes, the agent has analyzed the monitoring data, found a root cause and prepared a fix that waits for a human to approve and deploy [1]. Increasingly it opens the pull request for that fix as well [3]. In the other path nobody approves the incident at all. "The humans did the governance, set up the guidelines, gave some coaching to the agent, and then it went into auto mode to complete the entire workflow," Abdul Aziz said [7].
Microsoft says more than 3,000 of its own service teams use the agent [4], and that it has handled more than 1.8 million incidents internally, many mitigated in minutes [5]. Divide one figure by the other and the average team has put about 600 incidents through it; both numbers are stated as floors, so read 600 as an order of magnitude [15]. For any of that to transfer, the agent has to reach the same three inputs it correlates at Microsoft. Sanchit Mehta, one of the head engineers on the product, said it "starts analyzing telemetry and correlates things like blast radius, deployment changes, recent changes, any recent rollouts, to try to tell the engineers, 'OK, this is what is causing it'" [2].
InEight is the external example, and its starting position is the harder one. Correlating telemetry across tens of thousands of Azure resources there can take days or weeks. A ticket reporting slow performance without naming a product sends engineers to work out which of 14 products is affected and then check several observability and reliability tools [8]. On its first incident with the agent, InEight said the agent identified the affected product, traced the performance issue to its root cause and recommended scaling Redis. The DevOps team had been considering scaling the app service as a temporary fix [9].
The proactive claim is the one that leans hardest on the model. Mehta said the agent recently found the root cause of a change that broke synthetic tests as soon as the change reached the first region [10]: "It said, 'OK, this was an upstream PyPI package that broke your dependency; you need to add tests for it; you should roll back immediately; here's how you should go fix this.'" [11] He said that kind of monitoring is hard to handle with deterministic queries: "you need a level of intelligence to see when a large production payload is being deployed and if it has the potential to cause degradations" [12][13].
Vyom Nagrani, one of the head PMs, gave the strategic argument for it: "As code gets written more and more by agents, it's going to take another agent to operate it" [14].
The control surface is the list. The New Stack account does not say how those guidelines are expressed, who may add an action to them, or what review a change to the list gets [17]. A team adopting this writes down which mitigations may run at 3 a.m. with no one watching, and keeps everything else behind an approval. Restart, scale out, rollback and escalated change orders are a defensible first four [6].
What to watch
- Whether Microsoft documents the policy surface for the safe-action list: how it is scoped per resource, who can edit it, and what review an edit gets.
- Whether any external customer reports an autonomous share comparable to the more than half claimed for some internal Microsoft teams.
- Whether agent-authored pull requests for fixes ship generally, and what approval gate sits on the merge.