Skip to content

Leadership1 publisher3 min readPublished

Google's incident guide puts command outside the reporting chain

Google's Site Reliability Engineering site has published a primer on how the company runs an outage, from what is allowed to page an on-caller to who takes charge once one does. The command structure is borrowed from US emergency response.

The Board Room · Leadership desk

Illustration accompanying Google's incident guide puts command outside the reporting chain

What happened

  • Google's Site Reliability Engineering site has published a primer of the end-to-end incident response process it says it has developed and refined over the years.
  • The guide states that incident roles do not follow reporting chains and are instead assigned on knowledge and incident context.
  • Its preparation section lists four attributes of good alerting, including symptom-based coverage of user-facing functionality, and says alerts an on-caller cannot act on generate noise.

Compiled by The Board RoomSomething wrong?How this is made

Why it matters

  • decision Accepting this means deciding in advance that a director who joins the call is a stakeholder for the communications lead to brief, not the person running the response.
  • constraint The symptom-based rule limits what an engineer may page on: an alert tied to internal behaviour has to be justified as a quota-style cliff or be dropped.
  • cost Playbook upkeep, drills and training awareness take on-call engineers' time in quarters when nothing goes wrong.
  • capability Automating severity and scope analysis and suggested mitigations moves the on-caller's job toward judgement, and puts incident classification inside code that someone then owns.

A command structure that ignores the org chart only works if it is agreed before anything breaks. Google's guide says it directly: "Incident roles do not follow reporting chains and instead are based on knowledge and incident context" [6]. The incident commander coordinates the response, the communications lead issues stakeholder updates and takes inbound questions, and the operations lead stays on mitigation [5][3]. Any of the three may delegate tasks to other responders [5].

The lineage is explicit. IMAG is based on the Incident Command System, the US standard for responding to emergencies such as wildfires and earthquakes, and it keeps that system's three Cs: coordinate, communicate, control [3][4].

The alerting section is where the guide gets specific about what an on-caller is allowed to be woken for. It lists four attributes of a good alerting mechanism: timeliness, coverage of all key user-facing functionality, symptoms over causes, and actionability [7]. Alerting on service level objectives, the guide says, gets you the first three [8]. That leaves one of the four as a separate editorial judgement about each page [2]. On the third, the guide says alerts should be "based on end-to-end measures of customer/client experience, not based on a system's internal behavior" [17]. It carves out one exception: preventive alerts on internal metrics such as an approaching hard resource quota, where a system can go from 0% failure to 100% failure instantaneously [9]. Otherwise internal-behaviour alerts are discouraged because they do not map to user impact and are fragile, being bound to the implementation as it stood when the alert was written [10].

Preparation has to happen whether or not an outage does. Playbooks have to be current, and the guide notes that on-callers must actually know the playbooks and training material for them to help [11]. Google's named practice for that is the "Wheel of Misfortune" exercise, which also gives less experienced on-callers somewhere safe to learn [12]. Where response steps can be automated, the guide wants them automated, including severity and scope analysis, root cause analysis, and suggested mitigating actions, so on-callers spend their attention on problem solving [13].

The guide is process description without measured results: it does not publish detection or mitigation times [16]. Written by Google's own reliability engineers, it is not evidence about anyone else's outages. What it does supply is a set of commitments a management team can accept or refuse in calm weather, and refusing them is the default. The guide's demands fall on preparation, and the guide's own framing is that outages are inevitable in any sufficiently complex system [14].

Copying three role names takes an afternoon, but playbook currency and drill cadence have a calendar attached and decay [11][12]. And the guide's test for a good response asks that users, stakeholders and leaders be told what is affected, how bad it is, what workarounds exist, and when the incident may be mitigated and resolved [15].

What to watch

  • Whether Google publishes detection or mitigation figures that would let outsiders test the process against outcomes.
  • Whether teams that adopt the IMAG role names also retire their internal-metric pages; that half of the change costs engineers alerts they trust.
  • Whether drill cadence and playbook upkeep survive a quarter with a shipping deadline, since the guide's preparation claim rests on both.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories