Tailscale's 19 corrupt SQLite files show single-writer discipline is not corruption detection

Nineteen corruptions in six months is about one every nine or ten days ^[9][17]. Tailscale calls SQLite corruption highly unusual and not something you should meet in normal operation ^[16], and that can be true at the same time as the rate above, because the unit differs. Per shard, the odds stayed small, and most shards were never involved at all ^[15]. Across the fleet, something broke every week and a half ^[20]. The status page reports the fleet ^[15].

The first signal, last August, did not come from the process that owned the database. It came from a data pipeline reading the S3 backups, and the integrity check was run against the backup copy ^[8]. That pipeline uploads the entire database file every few minutes ^[7], so a corrupt file reaches the bucket before anyone has looked at it, while the single Go process holding the write lock keeps serving ^[5]. Detection latency was therefore the snapshot interval plus whatever schedule the downstream reader ran on ^[18]. Nothing in the serving path was looking.

That is the part that generalises, more than the WAL internals. Tailscale had done what the documentation asks: one process, exclusive access, the pattern SQLite is built around ^[5], on a database it has run as primary storage since 2022 ^[6]. The fault was in SQLite itself, in WAL reset ^[3]. Correct usage bought a great deal and it did not bound corruption.

With the cause unknown, two mitigations were available and only one was reachable. Recovery got faster across the incident series ^[13]; incidence did not fall until the bug was identified months later ^[1][2][19]. That is a reasonable response to an undiagnosed fault, and it also means the team spent half a year treating corruption as a recurring operational event, paid for by stopping the shard's control plane each time ^[12].

For anyone running shard-per-SQLite, the practical reading is arithmetic. PRAGMA integrity_check exists and is cheap enough that Tailscale ran it by hand the moment it had a suspect file ^[8]. The number of database files scales with the shard count, and each one is a separate object that either has a named owner checking it on a cadence you set, or does not. Inheriting detection from a backup consumer means your mean time to detect is a property of your analytics schedule.

One more thing kept the blast radius survivable: these databases hold tailnet and device metadata, never private keys or traffic ^[10], so the worst outcome in the early incidents was a handful of recent devices and configuration changes that had to be re-entered ^[11]. Devices already connected stayed connected, blind to changes, while anything arriving during the repair window could not join ^[14]. A shard-per-SQLite design holding state that cannot be re-typed would have produced a very different post.