A stale queue connection slips past the readiness probe unnoticed

The probe contract, read literally, explains the surprise. A liveness probe reports whether the process has deadlocked or hung badly enough to fail its own restart threshold, and a readiness probe reports only that the process is willing to accept traffic right now ^[3][4]. The startup probe exists so the kubelet does not evaluate either of those before initialization finishes ^[5]. That is three checks on the state of a process, none of which runs business logic ^[18].

A stale connection to a downstream queue walks through all three without touching any of them. In the author's account of the incident pattern, the readiness probe on the checkout service never checked whether the service could reach the fulfillment queue; it checked whether the process was up, and the process was up ^[10].

Each of these checks was built to answer a cheap, fast, narrow question rather than an expensive, slow, broad one, and the author's position is that the tradeoff is usually correct ^[11]. Cheap and fast has teeth. A check that re-dialled the queue every poll interval would be neither, and the kubelet keeps polling for the life of the pod.

What the material does not contain is a postmortem. The forty-minute checkout failure is introduced as the scenario that makes the argument concrete ^[8]. The one instance pointed at outside the article is the Network Loop in the Rack2Cloud Method's Day-2 operations guide, which the author says documents the same failure mode from the routing side, a misconfigured readiness probe sending traffic to a pod that cannot actually serve it ^[12]. The article counts six control points where a passing signal hides a real failure, and readiness is the row it develops in the text ^[14]. So the probe semantics are documented and the forty minutes is an illustration.

For the pattern to transfer to your cluster, two things have to hold at once. The dependency is established once at startup and never re-validated on the serving path ^[9]. And the probe endpoint does not touch that dependency ^[10]. Break either condition and the failure stops being invisible: a handler that opens the connection per request and returns a 500 when it cannot shows up in the error rate you already graph, not in the replica count.

Nothing in the orchestration layer was lying in this scenario. The rollout genuinely succeeded by every metric Kubernetes tracks ^[17]. The dashboard answered the question it was asked, accurately: it confirmed that replicas were ready to receive traffic, not that a checkout could complete. I would act on the mechanism and treat the forty minutes as a teaching example until someone publishes the timeline.