Build1 publisher3 min readPublished
Kubernetes' histogram KEP puts the migration switch in the Prometheus scrape config
KEP-5808 would expose Kubernetes component metrics as Prometheus native histograms alongside the classic bucket series, but only to scrapers that negotiate a protobuf format, and removing the old series is an explicit non-goal.
The Engineer · Build desk
What happened
- KEP-5808 proposes exposing Kubernetes component metrics as Prometheus native histograms, which use exponential bucket boundaries with automatic adjustment in place of predefined fixed boundaries.
- The KEP claims native histograms cut the time series count per histogram by roughly ten times, alongside improved query performance and finer-grained visibility into distributions.
- Rollback runs primarily through Prometheus-side configuration for Prometheus 3.x users, with the Kubernetes feature gate as the other route.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Clusters whose Prometheus scrapes the text exposition format get nothing new from the gate being on, so the migration is gated on scrape configuration and Prometheus version, not on the cluster upgrade.
- cost Teams pay for both formats during the overlap, and the promised storage saving only lands after they stop scraping or retaining the classic bucket series.
- decision Platform teams keep control of when to move alert thresholds and quantile queries off le-label buckets, because this proposal sets no removal date for the series those queries read.
- capability Latency differences that the default bucket set flattened into one bucket become separable, which matters most for the fast tail the current boundaries cannot resolve at all.
Content negotiation decides what arrives on the wire, not the feature gate. With `NativeHistograms` on, a component exposes histogram metrics in both classic and native form at once. The native form appears only when the client requests a format that supports it, such as PrometheusProto [3]. Two scrape-side prerequisites sit behind that: native histograms landed in Prometheus 2.40 and became a stable feature in v3.8.0 [8][2].
Pull the storage claim apart before anyone budgets against it. The KEP counts classic histograms as one time series per bucket plus `_count` and `_sum` [6], and its own example bucket set lists eleven boundaries in seconds [7]. That is thirteen series for one histogram [1]. A tenfold reduction [4] against thirteen lands near 1.3. A histogram carrying four buckets is six series under the same rule, and six cannot fall by ten without going below one series [2]. So the 10x figure transfers in proportion to how many buckets the histograms you actually keep already define, and Kubernetes exposes hundreds of histogram metrics across control plane components [5].
The accuracy problem interests me more. Under the default buckets, a request finishing in 1 microsecond lands in the same `le="0.005"` bucket as one finishing in 4 milliseconds. A 4000x difference in latency becomes indistinguishable, as the KEP puts it [7]. Everything between 1s and 2.5s also collapses into one bucket [7]. Exponential boundaries with automatic adjustment are the fix the KEP proposes [8].
Fixed-bucket dashboards survive this intact. The KEP lists removing the classic exposition format and removing existing histogram metrics as non-goals [9], and it says compatibility with existing monitoring infrastructure is maintained through the dual exposition strategy [1]. Queries keyed to `le` labels keep resolving. What moves is the location of the off switch: rollback is handled primarily through Prometheus-side configuration for Prometheus 3.x users, or via the Kubernetes feature gate [10]. If the monitoring team owns the scrape config and the cluster team owns feature gates, backing this out is two tickets in two repos.
The storage headline skips the cost during the overlap. While dual exposition is live and the scraper accepts a native-capable format, both the classic bucket series and the native series are being stored. The reduction only arrives once the classic buckets stop being scraped or retained [3].
On status, the signoff checklist requires the following before the KEP targets a milestone: approver sign-off on `implementable` status, documented design details, a test plan with SIG Architecture and SIG Testing input, and a completed production readiness review [11]. It also requires a minimum two-week window for GA end-to-end tests to prove flake free [12]. The text does not name a target release. Among the goals is to "Provide a safe, gradual rollout path with extended testing periods" [13].
What to watch
- A target milestone for the NativeHistograms gate, plus the graduation criteria it has to clear.
- The published global defaults for native histogram configuration, since they fix resolution for every component metric at once.
- Any later revision that moves removal of the classic exposition format out of the non-goals list.