Nineteen corruptions in six months: why a checksum mismatch deserves a defect hunt

An engineer at Tailscale, Alex Chan, has published a postmortem describing 19 separate corruption incidents in the SQLite databases behind the company's control plane over six months, eventually traced to a fault that had been sitting inside SQLite itself for at least 16 years ^[1][2]. What makes it worth an operator's time is not the exotic root cause but the default it breaks: corruption of one file on one machine invites a hardware shrug, and in this case a shrug would have kept the bug alive.

The architecture was conservative. Tailscale's control plane is split into shards, each with an SQLite database holding the configuration data for the tailnets on it, and a single Go process accesses that database exclusively ^[3]. That single-writer arrangement is exactly how SQLite is meant to be used, according to the writeup ^[4]. The backup pipeline snapshots the whole file every few minutes and uploads it to S3 ^[5]. It ran without incident from early 2023 until August of last year, when a data pipeline reading those backups reported an error; `PRAGMA integrity_check` against the backup confirmed the database was corrupt ^[6]. That is roughly eighteen months of clean operation before anything surfaced ^[2].

The cost was not abstract. When corruption was detected, the control plane process on that shard had to stop while the database was repaired or restored ^[7]. Early incidents ran over an hour of downtime per shard ^[8], during which tailnets on it lost the admin console and the API, and devices coming online could not learn about the network ^[9]. The payload was tailnet metadata, not private keys or traffic ^[10], so the blast radius was availability rather than secrets.

The first instinct was to treat it as a one-off: repair the database, investigate, find nothing ^[11]. Then it recurred, 19 times in total before the underlying bug was resolved ^[2] - an average of one every ten days ^[1] - including a six-week stretch between October and December with no incidents at all before they resumed ^[12]. There were no recent changes to blame; nobody had touched the low-level SQLite interaction code for years, and a careful review turned up nothing ^[13]. There were also no common factors: corruption was not tied to a single shard, customer, tailnet feature, time of day, or load level ^[14].

That last detail is the part I would put on the wall. Hardware failure correlates with hardware. A bad disk, a bad host, a bad rack shows up as the same machine appearing twice. Corruption that ignores shard, customer, feature, time and load has already told you it lives in code that every shard runs. Tailscale's response fits that reading: with no synthetic reproduction available, they deployed passive forensic telemetry into the live environment and waited for the next incident ^[15], and they bought a professional support contract from the SQLite developers to get direct access to the people who wrote it ^[16]. Given the defect was at least 16 years old ^[1], it predated their backup pipeline by something like fourteen years ^[3]. Nothing in code review was going to find it. The routine integrity check on a backup did.

One caveat on provenance: I am working from a secondhand summary by a Dhaka-based engineer who states plainly that everything in his account comes from Tailscale's published postmortem and the Hacker News thread, and that he has not seen the codebase ^[17]; the thread carried 1,141 points and 217 comments ^[18]. The copy I have breaks off mid-sentence on the second of the two decisive moves ^[19], so treat the methodology, not the mechanism, as the transferable part.

Worth watching: whether the fix lands in a named SQLite release and how far back the affected versions run, and whether other teams running single-writer SQLite start finding old corruption once they check for it.