Nothing was down, in the strict sense. The process stayed up and the endpoint answered; the queue filled with jobs that failed anyway [8]. That is the shape of all three incidents: a liveness probe asks whether the endpoint responds and gets a yes, because 401 and 403 are responses.
Add the three detection gaps and the week carried about 112 hours of degraded service: 60 hours on the Chatwoot inboxes, 14 hours of rejected calls, 38 hours of missing WhatsApp media [5].
The store side of the problem is the well-instrumented side. n8n can report that 114 credentials are referenced from 1,311 nodes across 260 workflows, an average of about eleven references each, and that a single Postgres connection accounts for 233 of them across 80 workflows [2][3][1]. None of the three failures was in that inventory [5]. The readers that broke were a bridge holding its own copy of the Chatwoot token per inbox app [6], a URL token derived from a signing key and stored at Telnyx as a webhook path [13], and a hardcoded `proxy_set_header X-Api-Key` line inside the `/api/files/` location of an nginx vhost [18].
The nginx one is the failure mode I would expect to recur anywhere config is layered. The tested procedure covered six locations, including the snippet that injects the key on the public media path [17]. The vhost's hardcoded header, left over from before the snippet existed, took precedence, so the seventh reader was the only one that mattered [18][6]. The snippet variable was correct the whole time and nothing read it, which is about the tidiest way a system can be wrong. No config diff finds that. Reading the value back through the path a client uses does: curl a real media path from inside the WAHA container through the public domain, and treat localhost 200 with public 401 as proof that an injection layer is holding an old value [20].
Two of these numbers are claims about his workload. The detection gap on the call service dropping from 14 hours to seconds depends on Telnyx exposing the configured webhook URL for read-back at boot, and on the application being able to see a 403 at the token layer rather than inside a provider's dashboard [16]. And 858 401s over 38 hours is about 23 an hour [4], which is why no rate threshold would have fired; the traffic in the call incident was six calls from four numbers [14]. At that volume the only detector is a human or a deliberate probe.
The fourth rotation is the part that transfers. The verification was written before the change, so that the new secret returns 200 and the old one returns 401, and then he waited for the two consuming workflows on 2-minute and 5-minute schedules to run [23]. The daily bridge check has the same shape and does not print the secret to do it: sha256 of the configured token compared against the first 16 hex characters stored in `access_tokens`, one line per app, non-zero exit if any fails [12].
One default is worth its own line. The secret rotated on August 25 leaked through n8n itself, because the webhook trigger node stores the full request headers, `X-Webhook-Secret` included, in execution data that anyone opening an execution to debug will read [21]. That makes every debugging session on a webhook workflow a rotation trigger, and the rotation then has to reach a Supabase config row, a file on the client's WordPress host, an n8n credential, and a keychain [22].
