Build1 publisher3 min readPublished
An unprivileged DaemonSet writes two files a root systemd loop reads every 15 seconds
Password rotation and OS rollout across a Kairos fleet come down to one kubectl apply. The privileged half of the design lives in a systemd service on the host, and one of the two files it reads names the next OS image.
The Engineer · Build desk
What happened
- A dev.to post describes rotating a login password across an immutable Linux fleet, or pushing a whole new OS version, by applying one Kubernetes ConfigMap instead of SSHing into each box.
- The OS is Kairos, where an upgrade writes the new image to an inactive A/B partition and flips the bootloader to it, so a bad upgrade rolls back instead of leaving a half-patched machine.
- The in-cluster half is an unprivileged DaemonSet pod with all capabilities dropped and allowPrivilegeEscalation set to false, reaching the host only through hostPath mounts on /oem and /usr/local.
- The privileged half is a root-owned systemd service, kairos-agent-watcher, which reads only those two files and then applies config, flips the A/B slot, runs the upgrade and reboots.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure The fleet's login password sits in a ConfigMap in kube-system, so read access to ConfigMaps in that namespace is read access to the credential, and RBAC on one namespace becomes the control on fleet login.
- decision Anyone adopting this owes themselves a provenance check on image references before the watcher accepts one, or accepts that a compromised pod chooses the OS every node boots next.
- constraint The mutation path runs through the cluster that the OS itself ships, so a node whose pod is not running cannot be rotated or upgraded this way and comes back into scope only when it rejoins.
- capability The split gives operators a way to do node-level root work from Kubernetes without a privileged DaemonSet, which is reusable for any host task that can be expressed as a file the cluster writes and a root service reads.
Applying the ConfigMap changes nothing on a node by itself. The pod reads the object and writes one of the two files it can reach, under /oem or /usr/local [6]. The root service on the host takes it from there on its next pass, and it passes every 15 seconds [7]. So a change lands on a given node between 0 and 15 seconds after the pod writes, 7.5 seconds on average [1].
The obvious way to cross that boundary was tried first and thrown out: a pod with the host's PID namespace, the host root filesystem mounted in, and all Linux capabilities intact [8]. "It works, and it's a trap," the post's author wrote [9]. The rule he drew from splitting it instead: "don't grant the network-facing thing the privilege; let it express intent into a narrow channel, and keep the privilege on the other side of that channel" [10].
Two files is a narrow channel by count, and the question is what those two files can express. The upgrade file carries an image reference, set in the sample command to image=docker.io/you/kairos-rhel96:v2 with version=v2.0.0 [5]. A Kairos upgrade writes that image to the inactive A/B partition and flips the bootloader to it [2]. The post calls the two writable files "the entire blast radius" and grants that an attacker who owns the pod can "write a config change or point at an upgrade image", while holding that they still cannot run an arbitrary command as root [14][11]. Naming the image every node boots next is root on every node at the next reboot [3]. For the two-file bound to hold, something upstream of the watcher has to decide which image references are acceptable.
The password path has a separate failure mode. The rotation command is a sed against live cluster state, which is an odd shape inside a system bought because nothing drifts [4]. If the current value is not the literal string kairos, the substitution matches nothing, kubectl apply re-applies an unchanged object, and the pipeline exits 0 [2].
The install-time piece has the same texture. A small service picks the smallest attached disk over 20GB that is not the boot media, so the OS does not land on a data disk or on the install media [12]. That protects the data disk only where the intended OS target is itself the smallest non-boot disk above 20GB on every machine in the fleet [5]. A 2TB data array is safe under that rule; a 40GB scratch disk beside a 120GB OS disk gets the OS. The author says that piece was not simple and is "the kind of thing you won't find in the docs", under a heading he calls "The one that actually hurt", and the available text ends at that heading [15].
What to watch
- Whether the host watcher gains any check on image provenance, such as signature verification or a registry allowlist, before it accepts an image reference.
- Whether the fleet login credential moves out of a kube-system ConfigMap into a Secret or an external source of truth.
- The disk-selection incident the post files under "The one that actually hurt", which would show where the smallest-disk-over-20GB rule breaks on real hardware.