An unsequenced work order defeated Google's redundant routers in 13 minutes

An optical link comes up only if both ends agree. The replacement part supported higher-density connectivity and was not compatible with the transceivers still in place in the data center fabric, so every fiber it was fitted to lost connectivity ^[5]. The fabric side kept its original optics throughout ^[5]. Service returned when hardware operations technicians on site identified the disconnected links and physically re-inserted the transceivers that had been removed ^[15].

The design around that work was sound. Redundant routers sit in separate rooms within the same building so that one router failing does not interrupt service ^[6], and the upgrade was meant to run one router at a time over several days, with traffic diversions at each step and verification between steps that the network had returned to a fully connected state ^[7]. The technician was handed the complete list of transceiver replacements across all routers with no instruction to sequence the work one router at a time ^[8]. The workflow also left out the human and software verification steps that were supposed to detect unintended disruption ^[9].

Every fiber path across the affected devices came out within 13 minutes ^[10]. Google says the speed and nature of the error kept warnings of incorrect action from reaching the engineer before connectivity was lost ^[11]. The one check that runs at fiber-by-fiber speed is the standing rule to halt if light is detected on a cable after unplug, and it was not followed ^[12]. The report's training action now describes it as "the mandatory requirement to verify whether light is on every unplugged fiber" ^[16]. Google also says it will finish deploying work stop alerting for actions that disconnect live fiber in its data centers ^[18].

Detection was immediate, through automated network loss monitoring and proactive probes ^[14]. Moving regional traffic away from the faulty zone took approximately 19 minutes in this instance, and Google intends to automate that down to around 5 ^[19]. The event ran 251 minutes ^[1]. The diversion automation therefore addresses about 14 of those minutes, close to 6 percent of the outage ^[2]. The remaining hours were spent locating dark links and re-seating optics in a building in Iowa's time zone, which is the part no orchestration system shortens ^[15].

The affected routers supported a fraction of capacity in us-central1-b and a small fraction in us-central1-f ^[4]. All zonal products were hit for some customers in those zones, and regional products drawing on capacity in that data center were affected until traffic diversions were fully in place ^[21]. In my view the finding that transfers is the coupling between a failover target and a maintenance workflow: if the second zone takes capacity from the same data center, one work order covers both. Google says most common regional network workflows moved to automatically sequenced orchestration in 2025 and that zonal workflows are still in progress ^[17].