The probe order and the retry ceiling: two config values that decide whether a rollout recovers

The regression that hides inside a correct manifest

Kubernetes offers three probes and they answer three separate questions: has the application inside the container started, is it ready to receive traffic, does it need to be restarted ^[11]. Skip them and the kubelet treats a container as Ready the moment the process exists, which is false for anything that spends its first seconds loading state before it listens on a port ^[12]. The startup probe exists to cover that window, and it only works because of an ordering guarantee: liveness holds fire until startup has succeeded at least once.

That guarantee broke. In the walkthrough ngrok published, the author reports finding and reproducing a bug introduced in Kubernetes v1.35.0 in which liveness probes fire before the startup probe has succeeded, and reproduced it on k3s, minikube and kind ^[1]. The demos in that piece run on webernetes, a partial port of Kubernetes to TypeScript carrying more than 100,000 lines of ported Go, with behaviour verified against k3s ^[16].

The consequence is arithmetic, not opinion. An httpGet probe counts 200-399 as success, runs every periodSeconds, and is allowed to fail failureThreshold consecutive times before the container is killed ^[13]. The stock numbers are failureThreshold 3, successThreshold 1, a 30-second termination grace period, and CrashLoopBackOff starting at 10 seconds and doubling to a five-minute ceiling ^[2]. The first crash restarts immediately; the second and later ones wait ^[17]. Follow the doubling and the backoff sequence runs 10, 20, 40, 80, 160, 300 seconds, so the ceiling arrives on the sixth wait with 610 seconds of accumulated delay behind it ^[1]. After that, a looping container gets twelve restart attempts an hour, forever ^[2].

Now put a workload in it that needs longer than three probe periods to become useful. Liveness kills it before startup can ever report success, and the container is graded on a check it was never meant to answer yet. Each of those twelve hourly attempts pays the full cold-start cost and then dies before serving anything, per replica, on hardware that is billed by the hour whether or not a request lands. DevOps'ish describes this as exactly the failure mode that makes a slow-starting service restart forever while everyone stares at the dashboard ^[3], and that is the honest description: the cluster is not reporting a failed rollout, it is reporting a container that keeps being restarted, which is a category most alerting treats as noise until someone reads the count.

The budget nobody set, in a file nobody owns

The second value is on the other side of the wire. GitHub CTO Vlad Fedorov's writeup covers a 7-hour, 47-minute outage on August 17 that took down github.com, authentication, Actions, the APIs, pull requests, issues and Copilot ^[4], which is 467 minutes of degraded platform ^[3]. The trigger was a critical infrastructure component in the Central US data center failing to scale under record traffic, with capacity pressure cascading until authentication began failing ^[5]. The part worth copying into your own postmortem template is what happened at the end: Copilot recovered last because a client-side retry loop pushed more traffic into a system that was trying to come back ^[6].

That retry loop is not in the failing service's manifest. It ships in a client, is deployed on someone else's release train, and can be perfectly reasonable at the load it was written for. The load it was written for is gone. GitHub reports adding over 3 million CPU cores and 120 petabytes of high-speed storage, with Azure going from 12% of platform load in May to 58% and carrying half of all Git operations ^[7][8], a rise of 46 percentage points and roughly a 4.8-fold increase in share ^[4]. Monthly commits went from 1.4 billion in April to 2.9 billion in August, with Actions runs at 115.4 million ^[9], which is 1.5 billion more commits a month, about 107% growth in four months ^[5]. An unbounded retry budget does not become a bug on a particular day. It becomes unsurvivable when the traffic behind it doubles, and this was already GitHub's second significant August incident after the Actions failure on the 6th ^[10].

Why both of these clear code review

They share a signature: the safety mechanism is present, configured, and not in the path that matters. The probes walkthrough demonstrates the same gap deliberately. A client sending requests straight to a pod's IP address bypasses the readiness mechanism entirely, so its requests keep failing throughout the container's startup window even while the pod reports not ready ^[15]. Readiness is only a valve if something consults it.

Stack the two failures and they interlock. A misordered liveness check prevents a container from ever declaring itself ready; a caller that does not consult readiness and does not bound its retries keeps hammering the thing that is trying to load. Both produce the same telemetry, which is high utilisation, high restart counts, and a request success rate that nobody has wired to a pager because the deployment technically progressed. Neither is visible in a model card, a benchmark table, or a capacity plan. One lives in a probe stanza and a cluster version; the other lives in a client library.

The tooling is starting to name the right failure class

There is a small, useful signal in what shipped alongside all this. Go 1.27, released 19 August 2026 ^[33], promoted the goroutineleak profile in runtime/pprof to general availability, which automatically detects permanently blocked goroutines ^[34]. The interesting thing is not the feature, it is the target. The class getting first-party instrumentation is stuck forever, not crashed. Crashes announce themselves. Restart loops and retry storms present as work.

The variable that is never in the comparison table

There is a precise analogue in a completely different fleet. Economists Christoph Siemroth of the University of Essex and Yeomyung Park of Sungkyunkwan University re-crunched 12 years of Backblaze data covering 443,156 drives and more than 1.66 million drive-years from 2013 through Q2 2025 ^[18]. Matching drives on age, capacity, form factor and operating temperature, they put HGST at roughly 41% of Seagate's failure rate, WD at about 52%, and Toshiba last at 107% ^[19], which is a 2.6-fold spread between best and worst once the confounders are held still ^[7]. The authors argue Backblaze's own quarterly reports stack old drives against young ones, because each manufacturer's fleet skews to a different era ^[20].

The number that carries over is the environmental one. Each additional degree Celsius of average drive temperature raised failure rates by 2.1% in their models, compounding to roughly 23% across ten degrees ^[21]. Against the 1.36% fleet-wide annualized failure rate Backblaze published for 2025 ^[22], a ten-degree hotter fleet would sit near 1.67% on that coefficient ^[8], though the coefficient comes from matched models rather than the fleet average, and the authors note their data is enterprise drives with no comparable workload data across manufacturers ^[23]. The shape is the point: an operating condition set in a facility, not a brand chosen in a purchase order, moves the failure rate by roughly a quarter. Probe periods and retry ceilings sit in the same place in the causal chain, and get even less scrutiny, because at least the temperature is on a dashboard somewhere.

Who eats it

Wasted capacity now has a visible market price. DDR5 prices have climbed sharply over the past year amid RAM shortages driven by AI data centers, pushing buyers toward DDR4 at lower but still inflated prices or onto the memory they already own ^[24]. When the memory market is that tight, hours spent restarting a container that never serves are not merely inefficient, they are hours bought at a premium.

And the person best placed to catch a retry budget in review is being pointed elsewhere. Gergely Orosz reports that unrealistic expectations are the leading cause of CTO and VPE jobs turning bad in 2026, including mandated engineering cost cuts of 20% to 50% and pressure on business results as AI coding bills rack up ^[36]. He spoke with almost 20 leaders on or considering a career break, and says six out of ten of the CTO-level people he asked are on the way out ^[35]. The cost-cutting mandate and the config review that would actually cut the bill are landing on the same desk, and per that reporting the desk is more likely than not to be vacated.

Two values, then. The cluster version that decides whether liveness respects startup, and the retry ceiling in a client you may not own. Neither is a property of the model, and neither will show up in a rollout that looks healthy from the outside.