A year of green backups hid 7 of 10 missing Android signing keys

A developer writing at dev.to reports watching a nightly backup job return a green check every day for more than a year, then asking how many restores had actually been performed: zero ^[1][2]. When a monthly restore drill was finally built, it found that 7 of 10 Android signing keys had never made it into the bundle, and five of the affected apps were live on the store ^[6][7]. The setup had two paths: a mirror of all source into a private repo, and an encrypted bundle of the things that cannot be recreated - notes, config, and signing keys - shipped nightly to a private channel ^[3]. Both were green daily. According to the author, green proved only that the upload finished; it never proved the contents were right or that the archive could be opened at all ^[4]. Two failures were found inside one month, and both stayed green throughout ^[5]. The first was a collector with three hardcoded paths. New apps kept shipping, the number of keys grew, the collector did not ^[6]. By discovery, only three of ten keys were being captured ^[1]. The consequence is not data loss in the abstract: without those keystores, updates to five published apps could never have been shipped again ^[7]. The second failure is the more familiar one. The mirror push failed seven days running, because a large binary hit the host's file-size limit, while the script printed a success line and returned exit code 0 ^[8]. Any monitoring layer is only as honest as the exit codes it reads. The fix is a scheduled job, not a runbook entry. Once a month it rebuilds the encrypted bundle without shipping it, decrypts it with the stored passphrase, extracts and counts the contents, checks the mirror is not stalled by reading the latest commit timestamp through the API, then deletes the scratch folder and the generated bundle ^[9]. The format is openssl-compatible AES-256-CBC with PBKDF2, SHA-256, 100,000 iterations ^[10]. The author deliberately avoided depending on the openssl binary on the grounds that a restore happens on a fresh machine with nothing installed, and Node's standard library is sufficient ^[11]. An empty zip decrypts perfectly well, so the pass criteria are numeric: fewer than 50 note files, a missing index, missing or invalid-JSON settings, missing recovery instructions, or fewer keystores than published apps all return a failure ^[12][13]. That last condition is the first incident encoded as a check, on the argument that a written post-mortem does not prevent a repeat but a failing check does ^[14]. Then the harder discipline: testing the test. A check that always returns green is described as worse than no check, because it distributes confidence while looking at nothing ^[15]. The judge function was mutated on purpose - threshold removed, ordering inverted, made to always return an empty list - and each mutation was confirmed to turn red ^[16]. Decryption got a control pair: the right passphrase must reproduce the sample, and the wrong passphrase must throw, with failure as the pass condition ^[17]. The first drill reported 495 note files, 18 signing keys, settings that parse as JSON, a recovery document present, 519 files totalling 2.43 MB, and a mirror committed the same day ^[18]. Eighteen keys against a collector that once knew three paths is a six-fold gap ^[2]. Worth watching: the drill itself now needs the same scepticism, since a year of nightly runs is roughly 365 green reports that tested nothing ^[3]. One reported trap is platform-specific - calling process.exit() straight after fetch on Windows can abort inside libuv while sockets are still closing, throwing an assertion failure after the logic has already succeeded ^[19].