Eleven minutes, three nodes, no humans: the real case for immutable node OSes

A three-node K3s control plane running Kairos Hadron upgraded itself to v0.4.0 in eleven minutes of wall-clock time on July 7 with zero human intervention after the pull request merged, etcd quorum never broken and no workload disruption, according to an account published on the CNCF blog on August 14, 2026 ^[1][2]. The number worth attention is not eleven minutes; it is that the two failures the author had to fix lived in the pipeline glue rather than in the operating system, which is exactly where risk goes when you stop patching nodes in place ^[5][15].

The substrate: three control plane nodes bootstrapped with OpenTofu, K3s in HA, Cilium as CNI, all provisioned as code before any workload ran ^[3]. Kairos Hadron is an immutable distribution built on A/B partition upgrades and cosign-signed images; it does not patch in place, it writes a new OS image to the inactive partition and reboots into it, and rollback is booting the old partition again ^[4]. The author's stated design constraint was to use as much CNCF tooling as possible to avoid vendor lock-in, and to get a reliable upgrade process rather than more features ^[16].

Six tools, one job each ^[6]. Gitea, self-hosted on its own Kairos plus k3s cluster with an Actions runner, holds every manifest, policy and upgrade spec ^[7]. Renovate watches quay.io/kairos/hadron for new tags and opens a PR bumping the image tag and the metadata.name of the upgrade custom resource ^[8]. A Kyverno ClusterPolicy rejects any upgrade CR whose image does not match quay.io/kairos/hadron:* ^[9]. Cosign verifies the signature against the upstream GitHub Actions OIDC identity ^[10]. ArgoCD sees the merge as drift and applies it ^[11]. kairos-operator cordons one node, pulls the image, writes the new A/B slot, reboots, waits for rejoin, then moves on ^[12].

Two details carry the run. The first is concurrency: 1. An earlier version had concurrency: 0, which the author read as "one node at a time" and which actually means all nodes at once; three control plane nodes rebooted simultaneously in a homelab test and quorum survived by luck, not design ^[5]. The second is that NodeOpUpgrade is one-shot: the operator marks it complete and never reprocesses it, so patching spec.image on an existing object does nothing and Renovate must bump metadata.name in the same commit to force ArgoCD to delete and recreate the CR ^[13]. The triggering diff was two lines, and the k3s version inside the tag did not change at all; only the OS image component moved, from v4.1.1 to v4.1.2 ^[14][18].

The second bug is the more useful one. Renovate's custom regex manager used extractVersionTemplate, a field that does not exist in the custom manager schema, so it silently did nothing: the image tag was bumped, metadata.name was left alone ^[15]. Under the one-shot semantics above, that combination produces a merged, policy-passing, signature-verified commit that upgrades nothing ^[20].

Three things to watch. Whether your config layer fails loud on unknown fields, because a silent no-op in a scheduler is indistinguishable from a healthy cluster. Whether admission policy scope matches intent: a glob on quay.io/kairos/hadron:* constrains the repository, not the tag, so tag correctness rests on Renovate and cosign ^[19]. And behaviour when a node does not rejoin, which a clean run with quorum intact tells you nothing about ^[22]. At three nodes the run averages under four minutes per node ^[17]; the author expects more workloads and more clusters to hang off this root, and that is where the eleven-minute figure gets tested ^[23].