Build1 publisher3 min readPublished
Fiber maintenance between us-west1 campuses cascaded into Spanner Paxos and the metadata server
Google's final report on the August 20 incident traces more than two dozen degraded products back to lost capacity on the links between data centers inside us-west1, after automated rerouting failed to move the traffic.
The Engineer · Build desk

What happened
- At 08:00 Pacific on Thursday, August 20, 2026, engineers began planned fiber-optic maintenance between data centers inside Google Cloud's us-west1 region.
- Automated rerouting mechanisms did not properly redistribute traffic to alternate capacity, traffic volumes exceeded what capacity remained, and the region began to congest.
- The resulting packet loss, throttling and latency reached Spanner Paxos consensus and the Unified Metadata Server, both of which showed significant latency spikes.
- Services sitting on Paxos and UMS returned timeouts and elevated errors, among them Cloud Storage, Cloud IAM, Persistent Disk and GKE, with control-plane and data-plane requests both failing.
- Google's final incident report lists more than two dozen affected products and features, including AlloyDB, BigQuery, Cloud Build, Compute Engine and Pub/Sub alongside roughly fifteen unnamed others.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A diagram that lists services is not a failure-domain map: spreading a workload across many Google Cloud products inside one region does not diversify the fiber path or the consensus layer they all sit on.
- exposure Control-plane calls degraded alongside data-plane traffic, so the standard in-event response of redeploying or scaling out through the API was reaching the same congested region.
- decision Teams whose failover target is another zone in us-west1 now have to decide whether that target shares the inter-campus links, because the only diversification this record supports is leaving the region.
- precedent The trigger was routine scheduled work on the physical layer, which means the same path is reachable again at the next maintenance window unless the rerouting behaviour changes.
Spanner's Paxos consensus and the Unified Metadata Server sit below the product catalog, so one latency spike in them surfaces under several different product names at the same time [5][6]. A service count measures the wrong thing. Nine products are named in the report and roughly fifteen more are not, which is how the tally reaches more than two dozen [8][15].
Maintenance began at 08:00 Pacific [2]. The incident was mitigated by 10:22 [7]. The gap is 142 minutes, matching the two hours and twenty-two minutes the account gives, so the published duration runs from the start of the maintenance window and not from the first customer-visible error [13][19][1]. A customer error window pulled from your own metrics will sit inside those 142 minutes.
The account points to Google status updates at 10:13 and again at 10:32 Pacific during the incident, and the text supplied breaks off before quoting them [12]. The second of those timestamps falls ten minutes after the mitigation time in the same report [14].
The propagation path is physical at the bottom and automated in the middle. The account puts the lost capacity at the physical layer and the failed redistribution in the rerouting mechanism above it [9][4]. Traffic volumes then exceeded what capacity remained, and the region congested [4].
For this to describe your exposure, two things have to be true. Your writes and metadata lookups have to land on Spanner and UMS instances inside one region, and your failover target has to sit somewhere that does not share those inter-campus links. The five-step chain in the report does not name a customer-facing service until the fifth step [10].
Google's narrative, as relayed by the dev.to account, stops at the finding that automated mechanisms did not redistribute traffic; it did not disclose why they failed or break impact down by zone [18]. The missing zone breakdown matters most for anyone who bought redundancy inside us-west1 by spreading across zones. If Paxos replicas span campuses, the quorum traffic rides the same links that lost capacity, and zone diversity inside the region buys less than the diagram suggests. The record supplied does not settle that, and I would not spend a migration budget on the inference alone.
The record does establish that multiple customer-facing services were exposed to a single inter-campus network condition, and that the relationship is invisible from any individual service name [17]. The account calls this Dependency-Blind Architecture: dependencies that exist in the environment but were never explicitly mapped, discovered when the cost of discovery is highest [16]. Cloud Build and IAM are both on the affected list, and control-plane operations failed alongside data-plane requests across multiple products in the region [8][6].
What to watch
- Whether Google publishes why the automated rerouting failed to redistribute traffic to alternate capacity.
- Whether a zone-level impact breakdown appears, which would show if in-region zone diversity helped anyone.
- Whether the next scheduled inter-campus maintenance window in us-west1 passes without congestion.