Firebase's zero-length flag entry crashed iOS apps at launch for up to six hours

The failure came in through configuration. Firebase delivers flags from its backend to the SDK inside each app, so the change that broke iOS clients was made on the backend ^[9]. Developers on the ticket, which passed 100 comments, asked Google to roll it back ^[8][9].

Rolling back the server did not reach phones that had already stored the bad response. Those apps kept serving the cached copy for about four more hours after the rollback, according to The Pragmatic Engineer ^[12]. The rollback itself took 52 minutes by the ticket's timestamps ^[20].

Gergely, who writes the Pragmatic Engineer newsletter ^[16], wrote that Firebase showed "shockingly poor incident management at odds with how Google itself usually deals with high-severity incidents" ^[15]. His timeline supports him on diagnosis speed. The outside developer posted the root cause 56 minutes into the outage ^[17]. That was 14 minutes ahead of Google's first word on the ticket ^[18]. He asked Google whether the ticket or its own monitoring raised the alarm, and had no reply when he wrote ^[7].

Google coined the term Site Reliability Engineer and wrote the SRE book ^[14]. Firebase's status page showed all systems green during the outage and after it ^[6]. The team closed the incident with a short report saying the outage was resolved ^[13].

I think the lesson for app teams follows from how this failure travelled. The bad state arrived as vendor config and then outlived the vendor's fix in a cache on the device ^[9][12]. A switch that helps has to sit in front of both. It also has to work when the app cannot stay open long enough to fetch anything, since these apps crashed on first opening ^[1]. On each launch, in order:

1. Check for a marker the app writes just before initializing the SDK and clears once start-up succeeds. If the marker is still there, the last launch died inside the SDK, so skip it this time. 2. Otherwise read a flag your own service controls from local storage, and initialize the SDK only if it is on. 3. After launch, refresh that flag from your own endpoint with a short cache lifetime.

The first step needs no network. A device that crashed once would skip the SDK on its next launch. The gate only works if the SDK stays idle until the app calls it. The newsletter does not report whether any affected team had a switch like this, or how one fared.