Skip to content

Build Not yet confirmed elsewhere1 publisher 2 min read Published

Informer-cache lag lets a status patch overwrite a condition set 49ms earlier

Operators of a European automaker's internal Kubernetes platform say a stale informer-cache read can overwrite a status condition set 49ms earlier. Their fix reads the object straight from the API server on every status update.

The Engineer · Build desk

How we use AI Send a correction

Illustration accompanying Informer-cache lag lets a status patch overwrite a condition set 49ms earlier
Generated illustration
One reconcile interval far outlasts the overwrite window Per the post: how recently set a condition a stale-read patch can clobber, beside the 20-second reconcile interval in its example

Key figures: a stale-read patch can clobber a condition set 49 ms earlier; the post's example controller reconciles every 20 seconds; a 20-second interval is roughly 400 times the 49ms window.

One reconcile interval far outlasts the overwrite window
FigureValueNoteClaim
Age of a condition a stale-read patch can clobber49 ms1
Example reconcile interval20 seconds9
20-second interval against the 49ms window, roughly400 timesderived from the post's two figures13

What happened

  • The team's first status helper read through the informer cache, and the uncached read went in as a fix during a live incident.
  • Every step embeds a shared BaseStep that carries both the controller-runtime client and an uncached reader.
  • All condition updates go through one shared helper, so the cache bug was fixed in one place instead of fifteen call sites.
  • The post says Kubernetes v1.36, from April 2026, lets controllers detect and skip reconciles that run behind the cache.

Why it matters

  • cost The helper reads before it knows whether the step has anything to change, so a no-op status update costs the API server one GET and no patch.
  • exposure The post's Environment chain has ten steps, each with its own Ready and Reconciling conditions, so one reconcile can issue a run of condition writes against a single object.
  • decision Teams on v1.36 now have a second remedy to weigh against paying a direct read on every status write.

The helper is short enough to read whole. Per the post, it runs four operations inside `retry.RetryOnConflict(retry.DefaultRetry, ...)`:

1. Deep-copy the object and `Get` it through `UncachedClient` into the copy [4]. 2. Deep-copy that fresh object as `base` [4]. 3. Call the step's `mutate` on the fresh object. It may return `ErrNoOp`, and the caller short-circuits [4][5]. 4. Patch status with `client.MergeFrom(base)` [4].

Because the read sits inside the closure, a conflict retry fetches the object again instead of reusing the first snapshot [4].

The 49ms figure is one fleet's observation: custom controllers on managed GKE at a European automaker [14]. For it to transfer, a second write has to land inside the cache's lag window after the first. Between reconciles the gap is small beside the interval. The post's example controller reconciles every 20 seconds [9], roughly 400 times the 49ms window [13]. The exposure is inside a reconcile, where the post says the cache can lag by tens of milliseconds [8].

In our view, one extra GET per status write is the right trade where several conditions are written per reconcile. The context is a controller that chains steps, as this one does. A controller that writes status once per reconcile has less to gain.

Skipping a behind-cache reconcile and reading fresh are different remedies. The post text breaks off mid-sentence in the bug section, so it does not say how a controller detects the lag on v1.36 or whether the authors dropped the uncached read there [15]. We would keep the uncached read until that is documented.

What to watch

  • The v1.36 documentation for behind-cache detection: how a controller sees the lag, and whether it makes the uncached read in RetryOnConflict redundant.
  • Any follow-up from the authors on API server read load after moving status reads off the informer cache.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence50
Adoption
Insufficient
Hype gap+10
Incentives
Insufficient
Confidence50
Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    Informer caches lag the API server by tens of milliseconds at scale; a stale read inside a MergeFrom patch can clobber a condition set 49ms earlier.

    Reported Supported Source: dev.to post, 'Kubernetes Controllers: The Cache-Staleness Bug at Scale'View cited source
  2. [2]

    The uncached read was not the original design but a live-incident fix; the first version of the status helper read through the informer cache.

    Reported Supported Source: dev.to postView cited source
  3. [3]

    Every step calls one function to update conditions, so the conflict-safe read-then-patch lives in one place; when the cache bug was found it was fixed once, not in fifteen call sites.

    Reported Supported Source: dev.to postView cited source

Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.

Add to ChatGPTAdd to Claude

Sources

1 independent publisher whose own reporting we read for this story.

  1. dev.to

    1 article · October 14, 2026

    Kubernetes Controllers: The Cache-Staleness Bug at Scale

Share your take

Let Clarity write the post for you.

Signed-in readers get a short post drafted on this story in the register they choose — narrative, analytical, or a direct position — editable to the last word before it goes anywhere. The share buttons at the top of this story work without an account.

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

Loading related stories