Kubernetes v1.35 refuses to start the kubelet on cgroup v1 nodes

Kubernetes now has two checks that stop a cgroup v1 node, and they run at different moments. On kubeadm-managed clusters the first is the SystemVerification preflight check, which comes from k8s.io/system-validators ^[4]. It returns an error during `kubeadm init`, `kubeadm join` and `kubeadm upgrade` when it detects cgroup v1 alongside a kubelet at v1.35 or later ^[4]. With an older kubelet the same check only warns ^[4]. The second check is in the kubelet's own startup, where `failCgroupV1` now defaults to true ^[1].

Clusters built without kubeadm hit only the second check ^[4]. The kubernetes.io FAQ is blunt about it: under the default configuration, a remaining cgroup v1 node fails during kubelet startup ^[5]. For anyone still below v1.35, it says to migrate every Linux node to cgroup v2 before upgrading, or plan to set the temporary override ^[5]. Chris Short, summarising the FAQ in his DevOps'ish newsletter, wrote: "Find those old nodes before your next upgrade does." ^[27]

The override is temporary

The override is one line in the kubelet configuration file ^[2]. With the config header from the FAQ's own examples ^[29], it looks like this:

```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration failCgroupV1: false ```

The FAQ calls the setting temporary. It says removal will follow the Kubernetes deprecation policy and points to KEP-5573 for the removal work ^[2]. The notice period has been long. cgroup v2 support has been stable since v1.25, and v1 support moved into maintenance mode in v1.31 ^[3]. v1.35 arrives ten minor releases after v2 went stable ^[30].

The FAQ describes the kubeadm preflight error and the kubelet override as separate checks ^[4][2]. It does not say whether `failCgroupV1: false` also clears the preflight error. We would prove that on a test cluster before an upgrade window depends on it.

Four floors, four components

Running the unified v2 hierarchy is the first floor. DevOps'ish's summary of the FAQ lists three more: kernel 5.8 or later; containerd 1.4 or later, or CRI-O 1.20 or later; and matching cgroup drivers on the kubelet and the container runtime ^[6]. Each floor belongs to a different component ^[6]. A node image that passes the kernel check still has to pass the runtime check. A current runtime still fails the list if its cgroup driver disagrees with the kubelet's ^[6].

There is a second, higher kernel line. For Memory QoS, the FAQ recommends kernel 5.9 or later, because on older kernels `memory.high` reclaim can trigger a known livelock ^[7]. From v1.36 the kubelet logs a warning when Memory QoS is enabled on an affected kernel ^[7]. A node on 5.8 clears the listed floor and still sits in the livelock range ^[34]. For fleets rebuilding node images anyway, we think 5.9 is the version to standardise on.

What the migration leaves alone

One known eviction problem survives the move. The kubelet treats `active_file` memory as not reclaimable, so an I/O-heavy workload with a large page cache can make the kubelet report memory pressure and evict Pods ^[8]. This is kubernetes/kubernetes#43916, and the FAQ says migrating to cgroup v2 does not by itself change that calculation ^[8]. The documented workaround is to measure first, then set equal memory requests and limits on the containers doing the heavy I/O ^[8].

v2 does add a memory protection model, one the FAQ says cgroup v1 cannot provide ^[9]. Memory QoS runs only on cgroup v2 nodes ^[9]. Enabling the `MemoryQoS` gate applies `memory.high` throttling to Burstable containers. The threshold comes from request, limit and `memoryThrottlingFactor`, which defaults to 0.9 ^[10]. The reservation policy defaults to `None`, which writes neither `memory.min` nor `memory.low` ^[11]. Setting `TieredReservation` maps Guaranteed Pod requests to `memory.min` as hard protection and Burstable requests to `memory.low` as soft protection. BestEffort Pods get neither ^[11]. The FAQ's opt-in example ^[29]:

```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration featureGates: MemoryQoS: true memoryReservationPolicy: TieredReservation memoryThrottlingFactor: 0.9 ```

The feature arrived as alpha in v1.22, was updated in v1.27, and is still alpha in v1.36 ^[9]. The project's general advice is to keep alpha features out of production. Teams that enable tiered reservation anyway are told to test it and account for hard-reserved memory first ^[12]. DevOps'ish also lists whole-container OOM kills, PSI metrics on by default and in-place Pod-level resizing among the v2 gains ^[13].

Swap needed separate accounting

Node swap reached General Availability in v1.34 ^[14]. A kubernetes.io post that benchmarks it gives two historical reasons swap was discouraged. Under cgroup v1, memory and swap shared one combined limit. Operators could not cap disk swap on its own, so a container's real memory use was hard to predict or isolate ^[15]. Kubernetes' swap support relies on cgroup v2's separate swap accounting ^[15]. The second objection was the latency of paging to spinning disks, which the authors say NVMe Local SSDs largely eliminate ^[16].

The v1.35 cutoff and the swap feature depend on the same kernel interface. A node that fails the first check could never have given swap an independent limit ^[15]. DevOps'ish describes the setup as a few lines of KubeletConfiguration with `LimitedSwap` and Burstable QoS on the pods ^[17].

The kernel build below 300 MB

The clearest test in the post is a Linux 6.1.1 kernel build ^[18]. Without swap, the smallest memory limit that avoided an OOM crash was 600 MB ^[18]. With swap routed to a Local SSD, the limit fell by half to 300 MB, and the build finished in 374 seconds against the baseline's 433 ^[19]. The gap is 59 seconds, about 14% faster on half the memory ^[31]. A constrained run beating its own baseline is a result we would want to reproduce before it goes into a capacity plan.

Then the limit went to 200 MB. The active working set moved into swap, I/O waits grew long, and execution time rose by over 40% ^[20]. If the reference is the 433-second baseline, the run took more than 606 seconds ^[32]. The step from 300 MB to 200 MB removed one third of the limit ^[33]. The post's authors drew the conclusion themselves: "This reinforces that swap serves as an insurance policy for burst memory, not a replacement for active RAM." ^[21] Short wrote it more plainly: "Swap is insurance for idle and bursty memory. It's not more RAM." ^[26]

The headline figures come from density. In gVisor sandboxes, headless Chrome went from 80 to 160 pods per node and Python sandboxes from 80 to 240 ^[22]. Those are gains of 2x and 3x ^[35]. The post reports density gains of up to 3x, "often with little or no latency cost" ^[23].

What would have to be true elsewhere

These are measurements of someone else's workloads on someone else's hardware. For them to carry over, a large share of resident memory has to be cold. Both tested workload types have that property. The post says earlier compiled objects sit inactive in memory while a CI job runs ^[24]. The agent sandboxes need a large footprint to start and run untrusted code, then sit idle waiting for the next prompt ^[25].

The swap device also has to be fast. The benchmarks ran on NVMe Local SSDs on GKE ^[17], and slow-disk latency was one of the two original reasons swap was discouraged ^[16]. The memory limit also has to stay above the active working set. The 200 MB kernel build shows what happens below it ^[20]. We'd expect a service whose resident memory is mostly hot to see little density gain and reach that slowdown almost at once.

The QoS class now drives more behaviour

Several of these features key off the same Pod fields. The swap setup DevOps'ish describes puts pods in Burstable QoS ^[17]. Memory QoS throttles Burstable containers at `memory.high` and, under tiered reservation, gives their requests only soft `memory.low` protection ^[10][11]. The page-cache workaround for #43916 edits the same request and limit fields ^[8]. For an I/O-heavy pod that also wants swap, we think the QoS class has to be chosen deliberately and tested against swap, throttling and eviction together. For sizing hard reservations, the kubelet exposes alpha metrics for a node's total `memory.min` and `memory.low` reservations ^[28].