A Kafka Streams auth cache kept a revoked API key working for four days

The check itself fails closed. `isKeyValid` reads one record from the pod's local RocksDB store and returns true only when the record exists and its status is 'active' ^[6]. A revoked key arrives as a tombstone: the key, with a null value ^[5]. For the flagged requests to pass, the three pods had to be holding an active record for a key that Postgres had already marked revoked ^[19][8].

The first theories went upstream and failed. A console consumer found the tombstone on api-key-status, committed and replicated ^[9]. So the fault sat downstream of Kafka. A stale edge cache was ruled out from the deployment topology, because the authz path has none ^[21].

Partitions 4, 7 and 9 were current to the high-water mark ^[10]. "Caught up with a wrong answer is a different bug than stuck behind," the post's author wrote ^[11]. Lag compares a committed offset with the log-end offset. A replica can reach the end of the log and still hold a record it should have dropped.

Nothing restarted the pods during the block. Their liveness probe was an HTTP ping against the gateway's own port, a test the gateway could pass while talking only to itself, so it stayed green ^[13]. The consumer's heartbeat stopped reaching the group coordinator. After session.timeout.ms, Kafka reassigned the three partitions to healthy members, and those members consumed the tombstone within seconds ^[14]. On the post's rounded figures, the block began about 120 hours before the fraud alert and lifted about 90 hours before it. The revoke, at about 96 hours, fell inside that window ^[20].

Static membership is the older fix in this story. The group runs with group.instance.id set, enabled after a rolling deploy forced a full rebalance and spiked auth latency across every pod, not just the ones restarting ^[15]. With it, a pod that drops out during a deploy keeps its partition assignment reserved ^[15]. "It was a good fix for that problem," the author wrote, adding that it "also meant nobody was watching for what happens when a pod goes quiet for reasons that have nothing to do with a deploy" ^[16]. In this outage the reservation did not hold. The partitions moved once the session timed out ^[14].

The rejoin is where the account stops. Kafka Streams keeps its store on local disk and checkpoints the last offset it processed. On reconnect, if that checkpoint is still within the log, it reads forward from there and skips a full replay ^[17]. The three pods' checkpoints were roughly 6 hours stale when the network came back ^[18]. The available text breaks off mid-sentence there, before it explains how the restore left the active record in place.

The latency result is real for this workload. Going from 40ms to 4ms is a factor of 10 ^[4]. The saving came from dropping a Postgres call that added a flat 40ms whatever the request needed ^[2]. It transfers to systems where a remote lookup dominates the auth check's p99. The cost is that every pod holds its own copy of revocation state. A revoke is complete only when each copy has applied its tombstone, and here the ticket closed on the Postgres write ^[7][8].