A developer drops a model into a repository and asks the platform team for an endpoint. The first pass is usually a pod with an accelerator attached and a service in front of it, and that works. The second week is when the awkward questions arrive: which model version is actually serving, whether the deployment can be reproduced, and why the first request after a restart is slow. CNCF's post lists model loading time as one of the signals platform teams may now need alongside accelerator utilization, scheduling and queue times, inference latency, throughput and endpoint health [10].
The two figures CNCF sets side by side are 66 percent of organizations hosting generative AI models running some or all inference on Kubernetes [1] and 7 percent of organizations deploying AI models daily [2]. Subtract and you get 59 points [13], but the bases are not the same: one counts organizations that already host generative AI models, the other counts organizations [13]. That makes the 59 points a rough indicator of scale rather than a precise gap. CNCF's own reading is that running AI on Kubernetes and having a platform ready to operate AI continuously are different things [4].
Running inference on the cluster is often taken as evidence that a platform supports AI. The second data point complicates that. The 2025 State of AI in Platform Engineering research cited in the post found 35 percent of platform teams still do not orchestrate AI workloads at all [3], which leaves 65 percent who do [12] and says nothing about how far that orchestration goes. Daily deploy frequency is the closer proxy for whether delivery has been automated, and it sits at 7 percent [2].
The mechanism CNCF describes is not exotic. Production AI pipelines are heterogeneous: data preparation, preprocessing, retrieval, orchestration and application logic may run on CPUs while training or inference uses GPUs or other accelerators, and a single workload can depend on several resource types [6]. That is a scheduling problem before it is a hardware problem, and Kubernetes is answering it with Dynamic Resource Allocation, a declarative way for workloads to request specialized hardware [7]. Making GPUs available is the easy half; folding heterogeneous compute into one resource model is the half that takes a quarter.
The delivery side is the same story with a new artifact. Code, build, test, deploy becomes code plus model plus configuration, then evaluate, deploy, observe, update [8]. The test CNCF proposes is whether a team can name which application, model and configuration are running and reproduce that deployment [9]. On top of that sits the golden path: model, resources, deployment, endpoint, observability, policy, where the developer states what the workload needs and the platform supplies a repeatable implementation [11].
Two questions cut through the gap CNCF describes. First, can a team name the exact model and configuration serving production right now and rebuild it from source. Second, when a workload was slow last week, can the team say where it was waiting rather than reading a GPU utilization chart, which CNCF notes cannot answer that on its own [15]. A yes to both marks an AI platform. A no to both still means real inference is running in production, the kind of thing that pages someone at 3am even without the rest of the platform in place. A split answer points to which half needs funding first, and the delivery half is usually cheaper than the scheduling half.
