Google's bad Firebase payload could crash iPhone apps for four hours after its fix

Someone taps an app on their iPhone on Monday evening and it shuts before the first screen appears ^[1]. They try again. The app has an analytics library inside it, and that library has just read an "incorrectly formatted payload" from Google ^[2].

Teams usually think of an analytics SDK as instrumentation, code that watches the app and reports back. Users saw something else: an app that would not start, and only the company that sent the data could fix it. From the first crash to a complete rollout of the fix took Google 2 hours and 11 minutes ^[1]. App teams were at least spared an emergency release. Nick Cooke, the Google software engineer who posted the timeline, wrote, "No SDK updates are required on your end to apply this fix." ^[7]

Crashes kept coming after the rollout. Cooke wrote: "Due to caching behavior, some app instances may still experience crashes for up to 4 hours after the rollout completed." ^[8] He expected the last of those to clear by 23:52 Pacific ^[9]. That puts 6 hours and 11 minutes between the first crash and the last one Google expected ^[2]. About 65 percent of that worst-case window came after Google's fix was fully out ^[3].

The coverage mixed up fix time and recovery time. 9to5Mac's report said the patch was fully rolled out "by around midnight Pacific Time" ^[10]. Midnight is close to Cooke's cache-expiry time, four hours after the rollout he described ^[9]. An incident plan that stops the clock at the vendor's "fixed" message would have closed the ticket up to four hours early ^[8]. Google's timeline does not say what the payload contained or how many apps crashed.

I'd sort every SDK in an app along two lines. One is whether it takes data from the vendor's servers while the app runs. The other is whether it runs before the first screen appears. An SDK that does neither fails only through a bug you shipped and could have tested. One that takes remote data but starts later can break a feature while the app still opens. One that runs at launch on local code can stop the app opening, but only through a version you chose to release. Firebase Analytics on Monday sat in the fourth box, with remote data read at launch ^[5].

Anything in that fourth box is an outside service that can stop your app opening. It needs the incident plan any outside service gets: a named vendor contact, someone watching the vendor's status updates, a support script for "the app won't open," and a recovery estimate that adds the vendor's cache lifetime to its fix time. That is my recommendation, and it is cheap. Starting the SDK after the first screen takes it out of the fourth box. The cost is whatever it would have measured in a session's first seconds.