Build1 publisher3 min readPublished
An unsigned subtraction in zfs_fillpage() zero-fills memory belonging to other processes
OpenZFS 2.4.4 and 2.3.9 fix the underflow, while the 2.2.11 release published the same minute does not, and systemd's CrashAction=freeze default holds a surviving host in that state until someone hits the provider's reset button.
The Engineer · Build desk
What happened
- The zeroed pages belong to whatever else was resident, so the processes that segfault are random ones with no relationship to the truncated file.
- systemd's CrashAction default of freeze leaves a crashed PID 1 hung indefinitely, blocking systemctl and leaving sshd to die with nothing able to restart it.
- OpenZFS shipped the fix in 2.4.4 and 2.3.9 on 21 August 2026, and Proxmox carries it as zfsutils-linux 2.4.4-pve1.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Sites standardised on the 2.2 series cannot patch in place, because the writeup's reading of the 2.2.11 tag finds no fix there; getting it means a minor-version move and whatever regression testing that implies.
- cost The systemd default decides how long the incident lasts. A drop-in one line long turns a night of downtime into a ten-second reboot.
- exposure Anything resident on the host is a candidate victim, including services that never touch the ZFS dataset involved, so the damage is not scoped to the workload that triggered it.
- constraint ECC hardware cannot catch this, because the kernel writes zeros through the ordinary path to a valid address; the symptom points at a hardware fault, and memory is the wrong subsystem to take offline.
Unsigned types have no negative values, so a subtraction that should land below zero lands near the top of the range instead. In OpenZFS before 2.4.4 and 2.3.9, `zfs_fillpage()` computed the read length as `io_len = i_size - io_off` on unsigned types [1]. The page fault has already fixed `io_off`. An in-place truncate drops `i_size` below it, the difference underflows to near 2^64, and `dmu_read()` zero-fills physical memory far past the page it was asked for [2]. A length near 2^64 bytes is roughly 18.4 exabytes [3]. The pages it walks into belong to whatever else was resident, so the processes that die have no connection to the file [4].
According to the writeup, the distinguishing detail in the dump is the length register passed to `memset`: a clean value, a tidy negative number, exactly minus another register [5]. Simultaneous segfaults across unrelated processes is also the classic signature of a failing DIMM, and that reflex sends you to scheduled downtime and memtest86+ [6]. Two counters settle it faster. Zeros in `ce_count` and `ue_count` under `/sys/devices/system/edac/mc/mc0/`, with no `Machine Check` in the kernel journal, mean memory as a subsystem is healthy [7]. While you are in there, `dmidecode -t memory` should show `Total Width` eight bits wider than `Data Width`, 72 against 64, because those are the physical bits doing the correction; equal widths mean ECC is not active whatever the spec sheet says [8]. ECC corrects bit flips in the DIMM; here the kernel walks the ordinary execution path and writes zeros into the wrong place [9].
The kernel survives the underflow [2]. systemd sets the length of the outage, and its `CrashAction` defaults to `freeze` [10]. A crashed PID 1 hangs forever: nothing restarts services, `systemctl` blocks, and sshd dies later with nobody left to bring it back [11]. From outside the host is indistinguishable from a powered-off one, with no SSH, no HTTP and no hypervisor UI [12]. The provider's panel reported power as normal and support had nothing to add [13]. The journal breaks the shutdown theory: the previous boot's entries run right up to a couple of minutes before the next boot's POST [14]. The last line before the reset was an nginx worker dying: "nginx[...]: [alert] ...: worker process ... exited on signal 11" [15].
That also moves the start of the incident. `last -x` reports `crash` without saying when it began, and the real start is the first segfault in the avalanche, which can be a whole night earlier than the reset [16].
The patched releases are 2.4.4 and 2.3.9, both published 21 August 2026, with `zfsutils-linux` 2.4.4-pve1 on Proxmox [17]. Version 2.2.11 went out the same minute and does not contain the fix, which the author says was verified by reading the code in the tag rather than the release notes [18]. Anyone pinned to the 2.2 line therefore has to move across a minor version to get the patch [19]. The separate one-line change is `CrashAction=reboot` in a drop-in, which turns a repeat from an overnight outage into a 10-second one [20].
What to watch
- Whether OpenZFS publishes a 2.2.x release that contains the zfs_fillpage() fix, and on what date.
- Whether distributions pinned to the 2.2 line backport the patch or move users to 2.3.9 or later.
- Whether systemd's CrashAction default changes, or distributions start shipping a reboot drop-in themselves.