Published · 5h agoScience2 min read
Kubernetes 1.35 restart loops: the 7h 47m has no source, but the amplifier does
Nothing in the probe documentation, the v1.35 release notes or KEP-4781 records an outage duration. What they do record is how one slow-loading model server drags the rest of a Service down.
Written for builders.See today for builders
What happened
- Kubernetes probes are diagnostics performed periodically by the kubelet on a container; based on probe results Kubernetes can restart unhealthy containers or stop sending traffic to containers that are not ready.
- Incorrect implementation of liveness probes can lead to cascading failures, resulting in restarting of containers under high load, failed client requests as the application becomes less scalable, and increased workload on remaining pods due to some failed pods.
- Liveness probes do not wait for readiness probes to succeed; to delay a liveness probe you must set initialDelaySeconds or use a startup probe.
- If a container usually starts in more than initialDelaySeconds + failureThreshold x periodSeconds, the documentation says to specify a startup probe checking the same endpoint as the liveness probe, with failureThreshold set high enough to allow startup without changing liveness defaults; the default for periodSeconds is 10s. If a startup probe fails, the kubelet kills the container and applies the restart policy.
- If a readiness probe returns a failed state, the EndpointSlice controller removes the Pod's IP address from the EndpointSlices of all Services that match the Pod.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
The amplifier in the upstream documentation is not the restart, it is what the restart does to the pods still standing. A liveness probe does not wait for the readiness probe to succeed [3], so a container doing slow initialization is answering health checks while it works. If it usually needs longer than initialDelaySeconds plus failureThreshold times periodSeconds, the kubelet kills it, and the documented default for periodSeconds is 10 seconds [4]. A model server is exactly the case the readiness section already describes: time-consuming initial tasks such as loading files and warming caches [12].
Then the second half of the loop. A failed readiness check makes the EndpointSlice controller remove the pod's IP from every matching Service [5], so its requests go to the replicas that remain, which get slower, which moves them closer to their own liveness tolerance. The documentation states the outcome without hedging: containers restarting under high load, failed client requests as the application becomes less scalable, and extra work on the pods that are left [2]. Client retries land on the same shrinking endpoint set.
The written remedy is arithmetic rather than new machinery. Put the long wait in a startup probe that checks the same endpoint as liveness, with a failureThreshold high enough to cover real startup and the liveness defaults untouched [4]; keep liveness on the same cheap endpoint as readiness but with a higher threshold, so a pod leaves rotation before it is hard killed [6].
Kubernetes 1.35 moves two things adjacent to this. In-place Pod resource updates are GA, so CPU and memory can be adjusted on a pod that previously had to be recreated [8], which removes a restart cause for limit-driven failures and none for probe timeouts. PreferSameNode is stable for trafficDistribution, strictly preferring local endpoints with fallback to remote [9], which concentrates traffic on whatever a restarting node's neighbours can serve. KEP-4781, which would stop a kubelet restart from changing container ready state, has alpha PRs and a feature gate name in its tracking issue and empty target-release fields [10].
As for the figure on the card: none of the three supplied sources contains an outage duration, an incident timeline, or the phrase model server [11]. 7h 47m is a property of one deployment's weight-load time and probe budget, and neither is in evidence here.
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Kubernetes probes are diagnostics performed periodically by the kubelet on a container; based on probe results Kubernetes can restart unhealthy containers or stop sending traffic to containers that are not ready.
- [2]
Incorrect implementation of liveness probes can lead to cascading failures, resulting in restarting of containers under high load, failed client requests as the application becomes less scalable, and increased workload on remaining pods due to some failed pods.
- [3]
Liveness probes do not wait for readiness probes to succeed; to delay a liveness probe you must set initialDelaySeconds or use a startup probe.
ReportedView cited source - [4]
If a container usually starts in more than initialDelaySeconds + failureThreshold x periodSeconds, the documentation says to specify a startup probe checking the same endpoint as the liveness probe, with failureThreshold set high enough to allow startup without changing liveness defaults; the default for periodSeconds is 10s. If a startup probe fails, the kubelet kills the container and applies the restart policy.
ReportedView cited source - [5]
If a readiness probe returns a failed state, the EndpointSlice controller removes the Pod's IP address from the EndpointSlices of all Services that match the Pod.
ReportedView cited source - [6]
A common pattern is to use the same low-cost HTTP endpoint for liveness as for readiness but with a higher failureThreshold, so the pod is observed as not-ready for some period before it is hard killed.
ReportedView cited source
Sources & coverage · 2 publishers
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- kubernetes.io5h agoLiveness, Readiness, and Startup Probes | Kubernetes
- kubernetes.io5h agoKubernetes v1.35: Timbernetes (The World Tree Release) | Kubernetes



