Skip to content

Build1 publisher3 min readPublished

GitHub bought three months of Actions headroom by rerouting a third of its jobs

GitHub's August availability report reads as a capacity argument rather than a bug report, with the Actions fix described as containment and the harder database failovers still ahead of it.

The Engineer · Build desk

Illustration accompanying GitHub bought three months of Actions headroom by rerouting a third of its jobs

What happened

  • A GitHub Actions incident that began at 15:22 UTC on August 6 ran for 10 hours and 42 minutes, with a large share of workflow runs failing to start or complete through the middle hours.
  • GitHub ran a production MySQL primary from Azure for the first time on August 11, and repeated the pattern with two more primaries on August 27.
  • Job-routing changes moved 33% of Actions jobs off a constrained production cluster onto spare capacity, which GitHub estimates added three months of headroom.
  • A 24-table authentication-core cohort left mysql1, GitHub's oldest shared database, taking about a million queries per second off its replicas, with query-hygiene changes removing another 120,000.
  • Read traffic hit new highs, with migrated services peaking at 60.4%, the monolith at 64.3% in Azure, and Git reads at 54%.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint The Actions reroute carries an expiry by GitHub's own estimate, so the durable isolation and capacity work has to land inside that window or the constrained cluster climbs back toward its ceiling.
  • decision GitHub says outright that it cannot completely eliminate risk while it invests in architecture and moves to Azure, which turns a fallback plan for CI into a live choice rather than something to defer until a fix date is published.
  • precedent Edge load-shedding levers were part of mitigating August's incidents, so shedding is now a first-line tool, and the cost lands on whoever sent the requests that get dropped.
  • exposure Shared-database health stays on the work list for the coming month, so the class of dependency mysql1 represents still carries production traffic after the authentication-core cohort moved off it.

The deployment on August 6 went to an internal Actions service that turns incoming events into jobs [2]. Rolling it out replaced pods, which briefly reduced capacity at one site and pushed the remaining sites past their limits as traffic shifted to them [4]. GitHub rolled the deployment back and confirmed its contents were not at fault [3]. So the trigger was pod count. The affected services were already running close to their capacity and concurrency limits, and a rollout that briefly reduced the number of running pods was enough to exhaust the available headroom [5]. In that state, the change that tips a system over is indistinguishable from the hundred changes that did not.

That is the frame to apply to the containment number. Peak cache CPU on the constrained Actions cluster went from 98% to 80% [13]. Eighteen points bought an estimated three months [13], which implies consumption of roughly six points of peak cache CPU per month at the platform growth GitHub reports [3][23]. That rate is a claim about GitHub's workload, not yours; it transfers to your CI planning only if your job growth tracks theirs. GitHub is explicit that the reroute is a near-term containment measure rather than the finish line, and that the August outage reinforced the need for more durable capacity and isolation [14].

The database reporting is the strongest part of the post. Eliminating about 59,000 seconds of wasted database work per hour [12] is 16.4 seconds of work per second [1], call it sixteen query threads busy continuously on nothing anyone asked for. Against that, the million queries per second lifted off mysql1's replicas [11] is 8.3 times the volume of the query-hygiene change [2], but volume is not the cost; wasted seconds are.

Reads show where the region move actually sits. The monolith peaked at 64.3% of reads in Azure while reads from migrated services peaked at 60.4% [10], putting the monolith 3.9 points ahead of the services built to move [4]. Git reads at 54% [10] trail the monolith by 10.3 points [5].

Writes do not split by percentage. A primary is one machine at a time, which is why the first Azure primaries are the milestone to track and the read shares are not. GitHub reports that client-observed write impact was minimal and that no customer impact occurred in the transition [7]. Those are two different statements, and the first is the one to plan against. Further primaries are scheduled over the coming weeks, increasing in complexity as GitHub learns from each failover [9], so the failovers with the most to go wrong have not happened yet.

Credit where it is due on the protection work: Git overload protection served 6.4% more traffic while improving 95th-percentile duration by 24% and maximum delay by 78% [15]. More traffic at lower tail latency is the rare change that is not a tradeoff.

The stated order is availability, then capacity, then features [21]. On August's evidence capacity is the binding term, and the coming month's list is more database primaries plus more automation around capacity management and auto-scaling [20].

What to watch

  • Whether the durable Actions isolation work lands before the estimated headroom expires, and what peak cache CPU reads in the next report.
  • The next report's account of the further primary failovers, and whether any of them produces impact GitHub describes as customer-visible.
  • Whether shared-database health work yields another mysql1-scale query reduction or only hygiene-scale gains.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories