Build1 publisher3 min readPublished
A cooldown set before the sudo check suppressed re-detection for 30 minutes at a time
A macOS daemon held 47GB of mostly compressed memory while reporting 264MB of RSS, and the script that spotted it had no permission to act. The repair is one sudoers line for one killall, verified after install.
The Engineer · Build desk
What happened
- A macOS dasd process pushed swap to 37GB and made Chrome's launchPersistentContext time out at 180 seconds, 21 hours into uptime, according to the guard script's own header comment.
- Every SNS lane in the author's pipeline finished a full day with 0 items, and nothing in the top-RSS view showed which process was responsible.
- dasd was already on the guard's auto-restart list, but the process runs as root and sudo -n failed on every pass, so the script only ever sent a notification.
- The script marks its cooldown before it checks whether the kill succeeded, so each failed attempt suppressed the next re-detection while the leak kept growing.
- The fix installs a single NOPASSWD rule for killall dasd into /etc/sudoers.d/lily-dasd as a root-owned, group-wheel, read-only file.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Because sudoers matches arguments and not just the binary path, this grant covers exactly one process name. A second memory hog needs a second line that someone has to write and review.
- exposure A syntax error in one file under /etc/sudoers.d takes out sudo for everything on the host. Validating before installing is what keeps the operator able to undo his own change.
- decision Anyone reusing this guard has to decide where cooldown marking belongs. Leaving it ahead of the outcome check means every unsuccessful attempt buys half an hour of not looking.
- capability launchd brings dasd back on its own, so the kill is a repair an unattended script can perform. No human needs to be awake for this particular failure any more.
RSS counts resident physical pages. The macOS memory compressor takes pages out of that count without returning them to the system, so a process can hold tens of gigabytes and rank nowhere in a top-RSS list. The header comment of the guard script records dasd at 264MB RSS with MEM 47GB and CMPRS 46GB [1]. The number the monitor sorted on was smaller than the real footprint by a factor of about 180 [1].
The threshold was not the problem. AUTO_RESTART_THRESHOLD_MB was 5120, and dasd was over it by roughly nine times [4][2]. Detection fired on every pass and the kill never ran, because dasd runs as root and the only non-interactive path to killing it is `sudo -n`, for which no NOPASSWD rule existed [5].
Then the ordering. `cooldown_mark` runs before the script inspects the result of the sudo call, and COOLDOWN_SEC is 1800 [6]. Each failure bought 30 minutes of not looking. Across 21 hours of uptime that caps the guard at 42 chances to act [3], and each one ended in a notification telling the operator to run `sudo killall dasd` by hand [5].
The grant is one line: `matsubara ALL=(root) NOPASSWD: /usr/bin/killall dasd` [7]. Sudoers matches arguments as well as the binary path, so the entry authorises killall against a process named dasd and nothing else [7]. The installed file is 53 bytes: the 52-character rule plus a newline [9][4]. Mode is 440, owner root, group wheel [8].
Two details make this more than a visudo one-liner. The rule goes to a temp file and through `visudo -cf` before installation, because one file with broken syntax under /etc/sudoers.d/ stops sudo working at all, including the sudo you would need to repair it [8]. After installation the script asks, as the target user, whether the grant is in effect: `sudo -u matsubara sudo -n -l /usr/bin/killall dasd`, with `exit 1` when that check fails [10]. The `-n -l` mode tests permission without running the command [11]. The author names the two accidents this closes as a typo that kills sudo, and thinking a rule was added when it is not actually live [12].
For the pattern to transfer, the target has to be supervised. dasd is a standard macOS daemon and launchd restarts it immediately, which is the author's stated reason for calling it a safe kill [3]. Aim the same rule at a service nothing restarts and the remediation is the outage.
One gap survives the fix. Nothing in the code shown moves `cooldown_mark` onto the success path [6], so a killall that exits non-zero for some other reason still starts the 30-minute cooldown before the outcome is known.
What to watch
- Whether a later revision of mem-hog-guard.sh moves cooldown_mark onto the path where the kill actually succeeded.
- Whether a macOS update renames or re-parents dasd, since the sudoers entry matches the argument dasd exactly and nothing else.
- Whether the guard starts ranking on compressed memory as well as RSS, so the next hog is visible before a day of jobs returns zero.