Informer-cache lag lets a status patch overwrite a condition set 49ms earlier
Operators of a European automaker's internal Kubernetes platform say a stale informer-cache read can overwrite a status condition set 49ms earlier. Their fix reads the object straight from the API server on every status update.
The Engineer · Build desk

Key figures: a stale-read patch can clobber a condition set 49 ms earlier; the post's example controller reconciles every 20 seconds; a 20-second interval is roughly 400 times the 49ms window.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Age of a condition a stale-read patch can clobber | 49 ms | 1 | |
| Example reconcile interval | 20 seconds | 9 | |
| 20-second interval against the 49ms window, roughly | 400 times | derived from the post's two figures | 13 |
What happened
- The team's first status helper read through the informer cache, and the uncached read went in as a fix during a live incident.
- Every step embeds a shared BaseStep that carries both the controller-runtime client and an uncached reader.
- All condition updates go through one shared helper, so the cache bug was fixed in one place instead of fifteen call sites.
- The post says Kubernetes v1.36, from April 2026, lets controllers detect and skip reconciles that run behind the cache.
Why it matters
- cost The helper reads before it knows whether the step has anything to change, so a no-op status update costs the API server one GET and no patch.
- exposure The post's Environment chain has ten steps, each with its own Ready and Reconciling conditions, so one reconcile can issue a run of condition writes against a single object.
- decision Teams on v1.36 now have a second remedy to weigh against paying a direct read on every status write.
The helper is short enough to read whole. Per the post, it runs four operations inside `retry.RetryOnConflict(retry.DefaultRetry, ...)`:
1. Deep-copy the object and `Get` it through `UncachedClient` into the copy [4]. 2. Deep-copy that fresh object as `base` [4]. 3. Call the step's `mutate` on the fresh object. It may return `ErrNoOp`, and the caller short-circuits [4][5]. 4. Patch status with `client.MergeFrom(base)` [4].
Because the read sits inside the closure, a conflict retry fetches the object again instead of reusing the first snapshot [4].
The 49ms figure is one fleet's observation: custom controllers on managed GKE at a European automaker [14]. For it to transfer, a second write has to land inside the cache's lag window after the first. Between reconciles the gap is small beside the interval. The post's example controller reconciles every 20 seconds [9], roughly 400 times the 49ms window [13]. The exposure is inside a reconcile, where the post says the cache can lag by tens of milliseconds [8].
In our view, one extra GET per status write is the right trade where several conditions are written per reconcile. The context is a controller that chains steps, as this one does. A controller that writes status once per reconcile has less to gain.
Skipping a behind-cache reconcile and reading fresh are different remedies. The post text breaks off mid-sentence in the bug section, so it does not say how a controller detects the lag on v1.36 or whether the authors dropped the uncached read there [15]. We would keep the uncached read until that is documented.
What to watch
- The v1.36 documentation for behind-cache detection: how a controller sees the lag, and whether it makes the uncached read in RetryOnConflict redundant.
- Any follow-up from the authors on API server read load after moving status reads off the informer cache.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence50
- Adoption
- Insufficient
- Hype gap+10
- Incentives
- Insufficient
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Informer caches lag the API server by tens of milliseconds at scale; a stale read inside a MergeFrom patch can clobber a condition set 49ms earlier.
- [2]
The uncached read was not the original design but a live-incident fix; the first version of the status helper read through the informer cache.
- [3]
Every step calls one function to update conditions, so the conflict-safe read-then-patch lives in one place; when the cache bug was found it was fixed once, not in fifteen call sites.
- [4]
In the printed helper, inside retry.RetryOnConflict(retry.DefaultRetry, ...), the code deep-copies the object, calls Get through UncachedClient into the copy, deep-copies the result as base, calls mutate (which may return ErrNoOp), and then calls Status().Patch with client.MergeFrom(base).
- [5]
The ErrNoOp sentinel lets a step's mutate function say nothing changed, distinct from a real error, so callers short-circuit cleanly.
- [6]
Each step exposes explicit Ready and Reconciling conditions.
- [7]
An Environment resource runs a chain of steps: EnsureFinalizer, ResolveClusterConfigRef, ComputeTargetNamespaceName, EnsureNamespace, ApplyResourceQuotas, ApplyLimitRanges, ApplyServiceAccounts, ApplyRoleBindings, ApplyNetworkPolicies, DeleteFinalizer.
- [8]
The informer cache can lag by tens of milliseconds within a single reconcile.
- [9]
In the post's example, a controller reconciling every 20 seconds floods the event stream with duplicates unless events are emitted only when visible condition state changes.
- [10]
Every step embeds a shared BaseStep that holds the controller-runtime client, an uncached reader, the scheme, a logger, and an event recorder.
- [11]
A step whose mutate returns ErrNoOp has already made one uncached Get and issues no Patch.
- [12]
The Environment chain has ten steps.
- [13]
A 20-second reconcile interval is roughly 400 times the 49ms window.
- [14]
The post's authors operate a major European automotive manufacturer's internal Kubernetes platform, running a fleet of custom controllers on managed GKE.
- [15]
Kubernetes v1.36 (April 2026) now lets controllers detect and skip behind-cache reconciles.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toKubernetes Controllers: The Cache-Staleness Bug at Scale
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Entities
- dev.toFollow
- KubernetesFollow
- controller-runtimeFollow
- Google Kubernetes EngineFollow