Topic
A measure of how fully GPU compute capacity is used versus idle, central to assessing AI infrastructure costs, financing risk, and data-center efficiency.
NetApp is pitching Novus, a file system built to exceed 100Tbps, on its own figure that data-starved GPUs can run below 30% utilization. Buyers adding GPUs have reason to measure what their clusters already use, though every number in the pitch so far is NetApp's own.
Reality
Amazon's new EKS managed addon reads Prometheus metrics from every model pod and sends each request to the one with room in its KV cache. The advertised 82% cut in first-token latency rests on a single 4.4-second baseline.
Reality
An American Banker opinion piece argues that GPU-backed loans audit serial numbers and liens while the compute that repays them goes unmetered, and it points at power finance, where lenders advance against settled megawatt hours.
Reality
The utilization data existed the whole time, recorded every second, in a central store that could not safely be opened to the team paying for the card. A CNCF write-up puts a tenant-aware proxy in front of it.
Reality
Two-thirds of an AI data centre's cost sits in the layer with the shortest life, and the chipmaker backstops only part of what that layer ends up worth. Somebody carries the difference.
Reality
One engineer's home lab tally: $1,400 a month for two A100s running 40% idle, against open-weight models he measured inside noise of GPT-4o. The break-even is real, and it sits high.
Reality