Skip to content

Leadership1 publisher3 min readPublished

Kubernetes v1.37 exposes its latency histograms twice over

Native histograms are beta and on by default, and Kubernetes keeps emitting classic buckets beside them, so the advertised 90 percent cut in time series depends on a later call about what to stop collecting.

The Board Room · Leadership desk

Illustration accompanying Kubernetes v1.37 exposes its latency histograms twice over

What happened

  • Kubernetes v1.37 graduates native histogram support for its metrics to beta and enables it by default, one release after the feature landed as alpha in v1.36 under KEP-5808.
  • With the gate on, the shared metrics package applies a bucket factor of 1.1 to every histogram, and the project puts the worst-case relative error for quantiles at roughly 5 percent.
  • Consolidating per-bucket series into a single series per histogram cuts time series by up to 90 percent, according to the announcement.
  • Because the change sits in the shared metrics subsystem, control plane and node components inherit it automatically, kube-apiserver's request duration metric among them.

Compiled by The Board RoomSomething wrong?How this is made

Why it matters

  • constraint The storage saving is unavailable while dual exposition runs, since a Prometheus still scraping le labels keeps every classic bucket series it had, plus the native series if it can read one.
  • decision Observability owners pick a side this cycle: move collectors onto the Protobuf payload and rewrite quantile queries now, or stay on classic buckets and defer the comparison until both formats are no longer live.
  • cost Resolution cost moves onto component memory. That is why the cap exists: kube-apiserver and the node components hold buckets that Prometheus no longer holds as separate series.
  • precedent Alert rules built on histogram_quantile over interpolated le buckets survive this upgrade on a compatibility layer, so the breaking event is whichever release retires classic exposition.

The 90 percent reduction in time series and the promise of an undisturbed observability stack do not arrive in the same release. With the feature gate on, Kubernetes components emit classic buckets alongside native spans, and the project says existing Prometheus servers, dashboards and alerting rules that read traditional text scrapes or `le` labels keep working unmodified [8]. A Prometheus that keeps reading `le` labels stores what it stored before. The saving lands when classic buckets stop being emitted or collected, and v1.37 does not do that [8].

That figure follows from the post's own example. Its sample boundary list holds 11 values [3][15], and each boundary is exported as its own time series for every label combination [4]. A native histogram is one series carrying spans, a zero threshold and scaling factors [5]. Going from 11 series to one is a 91 percent cut [16], consistent with the up-to-90 percent claim [6].

BucketFactor 1.1 makes each bucket at most 10 percent wider than the one before it, and the project puts the worst-case relative error for quantiles at about 5 percent, whether the operation took one millisecond or ten seconds [10]. That bound is roughly half the bucket width [17]. Classic quantiles come out of linear interpolation between fixed boundaries, and where the spans are coarse the post describes the estimation error as significant [13]. A p99 threshold tuned against interpolated values was tuned against that error, so the same threshold reads a differently computed number once its query moves to native spans [7].

MaxBucketNumber caps each histogram at 160 buckets, a limit the post ties to OpenTelemetry SDK recommendations for base-2 exponential aggregation and to protecting component memory under extreme outlier distributions [11]. Against the 11 boundaries in the classic example, that is roughly 14 times as many resolution points held inside a single series [18]. The components hold all of that in memory, and Prometheus counts it as one series [12].

The gate can be turned off, and dual exposition carries the old rules through an upgrade untouched [8], so for this quarter there is nothing here to schedule. The comparison is what has a deadline. While both formats sit in the same Protobuf payload [9], an operator can run the old query and the new one against the same request stream and see how far apart the two p99 numbers are before an alert depends on the answer.

The announcement does not say which Prometheus versions or scrape settings are needed to read the native spans [19]. The bucket options are applied inside `k8s.io/component-base/metrics`, so every component built on that package inherits them, including kube-apiserver and its `apiserver_request_duration_seconds` [12].

What to watch

  • A later release turning classic bucket exposition off by default. Both the storage saving and any rule breakage arrive with it.
  • Memory reports from large clusters running kube-apiserver histograms at up to 160 buckets each.
  • Whether non-Prometheus collectors read the native spans out of the same Protobuf payload.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories