Build1 publisher2 min readPublished
Kubernetes v1.37 Makes Native Histograms Beta and Enabled by Default
The NativeHistograms gate is on by default in v1.37, and components answer /metrics in both formats. What Prometheus stores depends on two per-job flags, one of which defaults to false and leaves classic dashboards with no data.
The Engineer · Build desk
What happened
- In Kubernetes v1.37 the NativeHistograms feature gate is Beta and enabled by default, where previously it had to be switched on by hand.
- With the gate on, components serve classic and native formats from the same /metrics endpoint, and the response depends on the Accept header Prometheus sets from the scrape job.
- A text scrape returns classic buckets only, while Prometheus requests the protobuf format, carrying both classic and native data, when a job sets scrape_native_histograms: true.
- The always_scrape_classic_histograms option defaults to false, so a job that sets only scrape_native_histograms: true stops ingesting the classic _bucket series that existing queries read.
- The documented migration runs in four steps: ingest both formats, rewrite queries and alert expressions, verify in staging, then switch classic scraping off to reduce storage overhead.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Because a text-format scrape still returns classic buckets only, the v1.37 default adds no ingestion volume in Prometheus until somebody edits a scrape job.
- decision Adoption is a change in two systems at once: a gate on the Kubernetes components, and per-job flags owned by whoever owns the Prometheus config.
- exposure Alert expressions are migrated in the same step as dashboards, so a job flipped without the classic flag leaves rules evaluating series that return nothing.
- cost The documentation's guidance to keep both formats during migration means paying to store both representations for as long as the migration lasts, and it puts no number on what turning classic off saves.
The classic format's cost is countable. The boundary set the documentation uses as its example has eleven boundaries, running from 0.005 to 10 seconds [6]. Add the `+Inf` bucket, the `_count` and the `_sum`, and one histogram is fourteen time series per label combination [8]. Multiply that by the label sets your own apiserver emits to get a figure for your cluster.
The page says native histograms provide "significant storage efficiency, improved query performance, and finer-grained visibility into distributions" [5]. It attaches no number to any of the three, so the only local estimate is a before-and-after series count taken on your own endpoint while both formats are being ingested [5].
Getting protobuf takes more than one flag. Prometheus sets the `Accept` header from the scrape job, so curling `/metrics` by hand returns classic buckets [11]. If your jobs pin `scrape_protocols`, the documentation says `PrometheusProto` has to be in that list [14].
The query rewrite is a suffix deletion: `histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))` becomes `histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))` [19]. That is workable during the overlap because the native data answers to the base metric name while the classic buckets keep the `_bucket` suffix, so an old panel and a new panel read different series out of the same scrape [13][19].
The page lists two ways to disable native histograms and says the Prometheus-side route is fastest, with no Kubernetes restart needed [20]. The gate is per component and each component's metrics are independent, so the exposition side can be turned on one component at a time [4].
Clusters still on v1.36, or any cluster where someone has explicitly disabled the feature, need `--feature-gates=NativeHistograms=true` on each component they want native metrics from [3].
What to watch
- Whether the v1.37 release notes turn the gate on for every listed component or stage it, given the gate is per component.
- The documentation carries no before-and-after series count from a real cluster, so watch for a published one.
- Whether installs older than Prometheus 3.x get an equivalent of the per-job scrape flags documented for 3.x.