Skip to content

Build1 publisher3 min readPublished

GitHub's cleanup job watched replica lag while the primary ran out of connections

GitHub's published post-mortem traces two hours of site-wide failure on September 13 to a data-cleanup job whose only brake measured replica lag, on the cluster holding permission data that nearly every authenticated request reads.

The Engineer · Build desk

Illustration accompanying GitHub's cleanup job watched replica lag while the primary ran out of connections

What happened

  • GitHub's status page records roughly 28 services degraded between 08:43 and 10:44 UTC on September 13.
  • At peak that morning, 96 percent of attempts to create an issue through the web interface failed, and account signup failures exceeded 90 percent.
  • GitHub's post-mortem says the offending job paced itself against database replica lag, which stayed healthy while load built on a primary nobody was watching.
  • Independent uptime trackers put GitHub's 30-day availability around 97 percent, which the dev.to write-up converts to roughly 21 hours of full or partial unavailability per month.
  • The author's conclusion is that almost nobody should fully leave GitHub, but everyone should spend one afternoon making leaving possible.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure Anything that mints tokens sits behind the same permission cluster, so a background job in GitHub's own data-cleanup path reached CI runs in repositories nobody touched that morning.
  • cost An exit costed from this material is costed from documented public pricing and published guides, so the engineer-hours of a real cutover are absorbed by whichever team runs one first.
  • decision Retries around token creation held the database saturated. A retry budget is one of the things a team decides before its next cron job shares a cluster with user traffic.

Start with the 70 minutes. The internal data-cleanup job began at 07:33 UTC [10]. The degradation window on GitHub's status page opens at 08:43 [1]. That gap is 70 minutes [1]. Through all of it the job's pacing check did what it was written to do, which was measure database replica lag [11]. The replicas stayed healthy. Load was building on the primary, which nobody was watching, and the job kept writing as the primary ran toward its connection limit [11].

The cluster it was writing to stores permission data, and that data is read on nearly every authenticated request on the platform [10]. Once the primary exhausted its connections, every request that needed it hung [12]. One default turned that into a site-wide failure: the database calls had no quick timeout, so web request handlers waited instead of failing fast [12].

The token path carried it further. 8.8 percent of GitHub App token requests failed, and about 4 percent of Actions workflows were affected because token issuance ran through the same broken path [3].

How that figure is measured matters before it goes into a risk model. Three percent of a 720-hour month is 21.6 hours, so the conversion the dev.to write-up makes holds [3][8]. The checks run from outside, and they weight partial degradation the same as a hard failure: the September 15 incident was Copilot code review jobs failing for hours because of latency in an internal caching service [4]. That bills a team whose reviews depend on Copilot and costs nothing to a team whose reviews do not.

The exit question is being asked because the list is long. The write-up counts four degraded weeks out of five [7], and the Hacker News outage thread passed 550 points with over 900 comments while an alternatives thread alongside it topped 600 [9]. On leaving, the post is explicit about what kind of document it is. Its author says the pricing of each option is documented public information and that the checklist is assembled from published migration guides; no production migration was run for the article [16]. A team budgeting a cutover needs engineer-hours. The published guides do not carry that number.

The option described in most detail is Codeberg: a German non-profit association running Forgejo, free for free and open-source projects, with a published stance against AI training on hosted code [17]. Gentoo started moving its public mirrors there in February 2026 [17]. Funding and fit both limit it. It is volunteer-funded infrastructure, and teams using it for private company code are not its target [17].

The general case is stated plainly in the post. "If you run any system where a cron job shares a database with user traffic, you have this exact bug sleeping in your codebase," its author wrote [18].

GitHub's remediation list is the reusable part: rate-limit background jobs against shared customer-facing databases by default, page on primary load as well as replica lag, bound retries, add request-level timeouts, and split the shared cluster so authorization data stops being a single point of failure [14]. Four of those five are changes to code and configuration a team already owns [4]. The cluster split is a data migration, and GitHub listed it last [14].

What to watch

  • Whether GitHub confirms the authorization cluster split has shipped rather than remaining a planned remediation.
  • Whether the independent trackers' 30-day availability figure moves once the bounded retries and request-level timeouts are in place.
  • A published cutover account that reports actual engineer-hours for moving private repositories.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories