Published · 2d agoBuild3 min read
GitHub's autoscaler watched the wrong meter, and auth, CI and Copilot fell together
The August 17 postmortem describes a sidecar that hit its concurrency limit while its autoscaler measured the host, then a VS Code retry bug that kept Copilot down hours after everything else returned.
Written for builders.See today for builders
What happened
- On August 17 GitHub experienced an outage lasting 7 hours and 47 minutes that disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues and Copilot.
- In a postmortem published August 20th, GitHub CTO Vlad Fedorov said traffic reached a new peak while a critical infrastructure component in the Central US data center failed to scale with it.
- GitHub's root cause analysis traces the initial failure to an Istio service-mesh sidecar that reached its concurrency limit; its autoscaling policy monitored the host service rather than the sidecar's capacity, so new resources did not arrive as traffic increased.
- The bottleneck spread until four HAProxy nodes exhausted their flow limits, after which the gateway authentication path developed severe latency and failures affecting dependent services.
- At the peak GitHub measured error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
GitHub published the root cause analysis for its August 17 outage on August 20, and the mechanism matters more than the 7 hours and 47 minutes it cost [2][1]. Two unrelated defects, one in an autoscaling policy and one in a VS Code retry path, degraded code hosting, authentication, CI and AI assistance in the same window [1][3][8]. The first defect is a measurement error. GitHub's analysis traces the start to an Istio service-mesh sidecar that reached its concurrency limit while its autoscaling policy monitored the host service rather than the sidecar's own capacity, so new resources never arrived as traffic climbed [3]. The saturated object and the observed object were different objects. Pressure then moved outward until four HAProxy nodes exhausted their flow limits and the gateway authentication path developed severe latency and failures [4]. Everything downstream of auth followed: GitHub measured peak error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads [5], with SAML and OIDC authentication, SCIM provisioning and Team Sync also affected [6]. The second defect set the headline duration. Slow responses from one internal endpoint activated a latent retry bug in VS Code, lifting Copilot Token Service traffic from a normal 7,000 to 9,000 requests per second to between 70,000 and 100,000 [8], an increase of roughly eight to fourteen times [23]. GitHub blocked token requests with HTTP 403 responses, reduced gateway retries and restored traffic gradually [9], and said scraping attacks against code-download endpoints complicated recovery [10]. Most services were back at 16:36 UTC, Actions stayed degraded until about 18:03 UTC, and the Copilot Token Service did not fully recover until 21:02 UTC [7]. Working backward from the published 7:47 duration, the incident began near 13:15 UTC, core recovery took about 3 hours 21 minutes, and the Copilot retry tail accounts for roughly 57% of the outage window [24]. A client-side retry loop, not the mesh, owns most of the number in the headline. GitHub says neither August incident was caused by a code or configuration change and that both were capacity failures [11]. The August 6 Actions incident ran more than nine hours, with 71% of workflow runs hitting infrastructure failures at peak and 75% of the remainder delayed by more than five minutes [12]. July's availability report counted eight service-degrading incidents [13], and in March GitHub had already acknowledged that rapid load growth, tightly coupled infrastructure and weak defenses against misbehaving clients were letting local problems spread [14]. The load is real: monthly commits went from 1.4 billion in April to 2.9 billion in August [15], roughly doubling [26], alongside about 130 million merged pull requests and 24 million new repositories a month [16]. The spending is real too, and did not prevent this. GitHub added more than 3 million CPU cores, 120 petabytes of high-speed storage and installed as much hardware as available power in its existing data centers allowed [17]. Azure now carries roughly 58% of platform load and half of all Git operations, up from 12% of platform load in May [18], a 46-point shift in three months [25], with GitHub aiming to move GitHub.com production traffic out of its own data centers by the end of 2026 and add a second Azure region [19]. Two committed changes are worth tracking: consistent retry limits, retry budgets and variable timeouts across service-to-service calls, and a review of lower-priority CPU and memory alerts for components that could fail during spikes [20]. The structural one is slower, namely isolating critical systems and removing shared dependencies between them [21]. Until that lands, treat GitHub auth, Actions and Copilot as one failure domain rather than three, and check what your own clients do when a token endpoint gets slow. Fedorov's post was his third public reliability update since March [27].
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
On August 17 GitHub experienced an outage lasting 7 hours and 47 minutes that disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues and Copilot.
ReportedView cited source - [2]
In a postmortem published August 20th, GitHub CTO Vlad Fedorov said traffic reached a new peak while a critical infrastructure component in the Central US data center failed to scale with it.
ReportedView cited source - [3]
GitHub's root cause analysis traces the initial failure to an Istio service-mesh sidecar that reached its concurrency limit; its autoscaling policy monitored the host service rather than the sidecar's capacity, so new resources did not arrive as traffic increased.
ReportedView cited source - [4]
The bottleneck spread until four HAProxy nodes exhausted their flow limits, after which the gateway authentication path developed severe latency and failures affecting dependent services.
ReportedView cited source - [5]
At the peak GitHub measured error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads.
ReportedView cited source - [6]
SAML and OIDC authentication, SCIM provisioning and Team Sync were also affected.
ReportedView cited source
Sources & coverage · 3 publishers
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- github.blogVlad Fedorov3d agoThe August 17 outage, and the work ahead
- thenewstack.ioFrederic Lardinois2d agoGitHub added 3 million CPU cores. It still couldn’t keep up with your commits.
- runtimewire.comRyan Merket2d agoGitHub blames capacity failure and retry storm for nearly eight-hour outage

