Product1 publisher3 min readPublished
An idle GPU hid for eleven days behind a Prometheus only the platform team could read
The utilization data existed the whole time, recorded every second, in a central store that could not safely be opened to the team paying for the card. A CNCF write-up puts a tenant-aware proxy in front of it.
The Product Desk · Product desk
What happened
- At a routine cost review, the month's GPU spend was the biggest line on the whole infrastructure bill, and nobody in the room could say whether the cards were being used.
- Every GPU's utilization had in fact been recorded every second for months, landing in one infrastructure-owned Prometheus that held every metric for every team across thousands of namespaces.
- When the team went looking, it found a card that had sat at zero percent utilization for eleven straight days, allocated and powered on, and invisible to the team that owned it.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- constraint Loosening permissions is not available as a fix, since any read grant on the central store is a fleet-wide read grant, so tenant visibility has to be bought with new plumbing rather than a role change.
- cost Those 264 hours of zero-utilization time land on a namespace budget whose owner has no query path to check it, which is the cheapest kind of waste for an organisation to keep paying every month.
- capability Once a curated slice is remote-written into a Prometheus the tenant owns, an idle allocation can trip that team's own alert instead of waiting to surface on a cost review slide.
- decision Platform teams have to choose between continuing to curate what each tenant can see and handing that to a declarative object, which means accepting whatever metric requests tenants write into it.
The card was allocated and powered on, and the health checks stayed green for the duration [5][6]. Nothing in that chain asks whether a GPU did any work; a green check confirms the card is alive, not that it is busy. Eleven days at zero comes to 264 hours of scheduled, energised hardware returning nothing [14], charged to a team that had no way to run the query that would have shown it [5].
The metric existing does not mean anyone can watch it. The store described in the CNCF post has one access setting: according to its authors, a Prometheus query endpoint is not namespace-aware, so a tenant who can run one PromQL query can run any query, including one that reads another tenant's request rates or capacity plans [7]. Read access is all or nothing, and the people holding it are the ones whose budget is not charged for the namespace [4].
The second wall is load. That Prometheus already scrapes and stores series for the whole fleet, and the post's argument is that pointing a few hundred engineers and their ad-hoc queries at it makes the store everyone depends on buckle, one expensive range query at a time [8].
The fix runs on existing infrastructure [15]. Authenticate the caller, restrict every query to that tenant's namespace below the query language so a clever PromQL string cannot climb out, and optionally copy a curated slice into a small Prometheus the tenant owns [9]. Authorization sits in kube-rbac-proxy behind Nginx, and the proxy finds backends through the Kubernetes API, fans the query across healthy ones, and filters what comes back [11]. Delivery runs the other direction: the proxy collects each tenant's curated metrics and remote-writes them into that tenant's own Prometheus [12]. The part worth copying is the paperwork. A MetricAccess custom resource lets the team declare which metrics it wants, on the reasoning that nobody is hand-curating metric lists across thousands of namespaces [10][13].
The post never puts a number on it. It says multiplying one idle card across a fleet turns "we're not sure" into real money every month, and it puts no figure on the wasted card and no count on how many others were idle [16]. The mechanism belongs with the platform team; the arithmetic belongs with finance.
The same test works on any expensive line item: the three costliest resource classes, and for each, whose budget it hits and who would notice within a day if utilization went to zero. Where those are different people, another panel in the platform team's Grafana will not close the gap, because the platform team is not the one being charged. A tenant-scoped query path will, and the bill for it is a proxy tier to operate and, if you take the delivery step, a small Prometheus per tenant [9][11].
What to watch
- Whether the proxy and the MetricAccess resource get published as installable open source rather than remaining a design described in a blog post.
- Whether the team ever publishes a fleet-wide idle rate or a cost figure, which is the number a finance partner will ask for.
- Whether tenants' own alerts start catching idle allocations before the next cost review, given that the per-tenant Prometheus step is described as optional.