BuildNot yet confirmed elsewhere1 publisher2 min readPublished
DCGM's per-device default leaves Kubernetes platform teams to meter shared GPUs themselves
NVIDIA's DCGM exporter gives every pod sharing a GPU the same reading, so three pods sum to 300% of one card, The New Stack reports. Platform teams sharing GPUs must build their own per-pod metrics and cost-split rules.
The Engineer · Build desk
Three key figures. If three pods share one GPU, all three look like they use the whole thing and the sum is 300% of a GPU. That is 200 percentage points more than the card has. Separately, the usual GPU utilization metric can read 100% while compute is barely doing anything.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Summed use when three pods share one GPU | 300% of one GPU | 4 | |
| More than the card has | 200 percentage pts | 17 | |
| Usual GPU utilization metric can read | 100% utilization | while compute is barely doing anything | 10 |
What happened
- With CPU and memory, the kernel already tracks usage per container, so a per-pod number is close to free, according to The New Stack.
- Kubernetes sees a GPU as a whole-number resource, nvidia.com/gpu: 1, with request and limit required to match and no overcommit at the scheduler.
- Base CUDA images ship NVIDIA_VISIBLE_DEVICES set to all, so a pod can see every GPU on the box if nothing overrides it.
- The usual GPU utilization metric records only that a kernel ran during the sample window, so a card can read 100% while its compute sits nearly idle.
- NVML reports memory held per process, and each process can be traced to its pod and team, though the totals will not quite match the card's.
Why it matters
- cost Whoever owns chargeback for shared GPUs has to build the attribution before the first invoice can be split, on a line the piece calls the most expensive on the bill.
- constraint The request-versus-usage rightsizing loop has no input at the start. A team must first work out how much of a card a workload got before it can talk about waste.
- exposure A bill keyed to what the scheduler allocated inherits any gap between that record and what ran on the hardware, and closing it takes a process-level check.
- decision Each team has to pick its utilization signal per workload (compute occupancy, memory footprint, bandwidth or power) before it builds dashboards on the headline number.
The trigger is sharing. NVIDIA's DCGM exporter reports at the physical device level by default [2], and the piece says that is fine when one pod has a GPU to itself [13]. Under time-slicing or MPS, the driver keeps one counter set for the entire card. DCGM then has no per-pod data to attribute [5] and hands each sharer the same value [3].
The sum overshoots the card by 200 percentage points [17]. The values still plot nicely on a graph, the piece says, and teams build full dashboards and reports on numbers that are wrong and hard to notice [18].
The scheduler is no help. The fractional sharing that inference needs happens beneath it, through time-slicing or MIG, and Kubernetes requests do not fit neatly onto that sharing [7]. The piece states the general case this way: "Almost every piece of visibility you take for granted on CPU and memory, GPUs make you build yourself." [16]
A bill split on NVML's per-process memory has its own problem. vLLM and frameworks like it claim most of the card as soon as they launch, needed or not [11]. In our view, a split by held memory then charges a vLLM pod for what it reserved at startup. That may be the right policy for a shared cluster. It is still a policy the platform team has to write.
The source is a practitioner explainer on The New Stack, and it reports no measurements. It does not say how much spend ends up misallocated. The 300% case needs pods sharing a card through time-slicing or MPS [3].
What to watch
- Whether NVIDIA's exporter adds per-pod attribution under time-slicing and MPS, where the piece says the driver exposes one counter set today.
- Any measured data on how much GPU spend gets misallocated by device-level readings. The New Stack piece offers none.
- Runtime configurations that honor NVIDIA_VISIBLE_DEVICES from the container, the case where allocation records and hardware use diverge.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence50
- Adoption
- Insufficient
- Hype gap
- Insufficient
- Incentives
- Insufficient
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
With CPU and memory, the kernel already tracks usage per container, so getting a per-pod number is basically free.
- [2]
NVIDIA's DCGM exporter reports at the physical device level by default.
- [3]
When a GPU is shared across pods with time-slicing or MPS, DCGM gives every pod sharing that GPU the same value.
- [4]
If three pods share one GPU, all three look like they are using the whole thing, and the sum is 300% of a GPU that does not exist.
- [5]
With time-slicing or MPS, the driver exposes a single set of counters for the whole GPU, so DCGM has no per-pod data to attribute and cannot distinguish which workload consumed what.
- [6]
A GPU shows up to Kubernetes as a whole-number resource, nvidia.com/gpu: 1; request and limit have to match and there is no overcommitting at the scheduler.
- [7]
The fractional sharing needed for inference happens underneath the scheduler through time-slicing or MIG, and Kubernetes requests do not fit in neatly.
- [8]
Base CUDA container images ship with NVIDIA_VISIBLE_DEVICES set to all; if nothing overrides it or the container runtime honors the env var from the container, a pod can see and grab every GPU on the box.
- [9]
What the scheduler recorded as allocated and what ran on the hardware can diverge significantly; getting the real number means looking at the process level and lining it up against what Kubernetes thinks it did.
- [10]
The usual GPU utilization metric tells you a kernel was running on the GPU for some slice of the sample window, not how much compute or memory bandwidth was used; GPUs can sit at 100% utilization while compute is barely doing anything.
- [11]
Frameworks like vLLM grab most of the GPU memory at startup whether they need it or not.
- [12]
NVIDIA's NVML library reports, per process on a GPU, how much memory it is holding, and each process can be traced back to the pod and team that owns it; the numbers are dependable but will not quite add up to the card's total.
- [13]
Device-level DCGM reporting is fine when one pod has a GPU to itself.
- [14]
The basic rightsizing premise of comparing requested against used and reclaiming the difference barely exists for GPUs at the start; you must work out how much of a GPU's capacity a workload got before you can have the waste conversation.
- [15]
Teams should work out early which signal means something for their workloads: compute occupancy, memory footprint, bandwidth, or power.
- [16]
Almost every piece of visibility you take for granted on CPU and memory, GPUs make you build yourself.
- [17]
Three pods each reading 100% of one physical card sum to 300%, which is 200 percentage points more than the card has.
- [18]
The numbers still plot nicely on a graph, but teams build full dashboards and reports on values that are blatantly wrong but hard to notice.
- [19]
The GPU is the most expensive line on the bill.
Sources
1 independent publisher whose own reporting we read for this story.
- thenewstack.ioGPUs on Kubernetes are bring-your-own-everything, starting with the bill
1 article · October 12, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.