Skip to content

Build1 publisher2 min readPublished

launchd's four-entry PATH stopped the publish step while the log kept reporting success

Over four days one daily publishing job failed three separate ways in the place it actually runs, and every one of those failures had already passed a hand-run test in the author's own interactive shell.

The Engineer · Build desk

What happened

  • The first version was started with nohup from /tmp, and when the machine restarted the four scheduled items riding on it were gone, with no error recorded anywhere.
  • Rebuilt as a launch agent, the job died at line 59 on node: command not found, because launchd gave it PATH=/usr/bin:/bin:/usr/sbin:/sbin and node is in /opt/homebrew/bin.
  • After the PATH fix the log went quiet for three hours while the process stayed alive, sitting in a single sleep 900 that ps reported with an elapsed time of 02:55:52.
  • The current version does one pass and exits under StartInterval 900 with RunAtLoad, and launchd fires it on wake if the interval elapsed during the suspend.
  • launchctl list showed the job with no StartInterval key, while launchctl print on the same label reported run interval = 900 seconds and runs = 4.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure The failure that outlasts all three fixes is the half-working one: a job whose Python step exits cleanly every fifteen minutes satisfies any monitor watching exit status, and publishes nothing.
  • decision Verification has to move to the target environment. A hand-triggered run from an interactive shell certifies the shell, and all three of these fixes passed that way before failing in place.
  • constraint A health check that greps launchctl list for a key will report a loaded schedule as missing. That is enough to send an operator rewriting configuration that already works.

The same loop also runs a reconciliation step written in Python, and python3 is in /usr/bin, so that step kept working after node stopped resolving [9]. The launch agent set KeepAlive true, so a job that dies comes back [5]. The log filled with successful runs every fifteen minutes while no article was published [9]. "A partly working automation is harder to notice than a dead one," the author wrote [10].

The fix is an export PATH line that puts /opt/homebrew/bin and /usr/local/bin in front of the four directories launchd supplies [11]. It went into every script the scheduler touches, not only the one that broke, because another script can invoke the same work from a different parent [11].

The third failure was timing. One sleep call had held for 10,552 seconds where 900 were intended, about 11.7 times the interval, so roughly ten passes never happened in that window [13]. The loop was not broken and was not stuck on a network call. A sleep does not necessarily advance while the system is suspended, and each suspend adds an unbounded gap to a while true; do work; sleep N; done loop [14].

The author also deleted the old looping script instead of leaving it next to the new one. The reason given was that two entry points for the same job means never being sure which one is running [17].

Then the inspection tooling got in the way. launchctl list does not print every key it holds, and launchctl print does [20]. "I was one command away from concluding the schedule had silently failed to load and rewriting it again," the author wrote [21].

There were three fixes. All three were correct, and all three were verified in an environment the job would never run in [22]. The launch agent was tested by triggering it once by hand. The loop was tested by watching a single iteration, and a single iteration is exactly the case where a sleep that does not advance looks identical to one that does [26]. The four PATH entries are a property of that machine and that install prefix. What transfers is the procedure [7][8].

What the write-up asks for is a run of the same logic in the new environment. Look for the environment's own failure modes: what the PATH is, what happens across a reboot, what happens across a suspend [25]. In the author's words: "When you change where something runs, run it once in the new place before believing anything you knew about it" [23]. The author calls that rule narrow [24].

What to watch

  • Whether the StartInterval version survives a multi-day suspend without a missed publish; the account covers four days.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories