Build1 publisher3 min readPublished
Codex's day-long outage exposed the missing mutex in a launchd file queue
Offset start times kept a shared file queue from colliding for months. One day-long Codex outage made every launchd lane start at the same instant, and the pickup scripts had no mutual exclusion to fall back on.
The Engineer · Build desk
What happened
- Codex spent September 17, 2026 returning usage_limit_exceeded, and 54 sessions piled up under that day's session directory, every one of them terminated with the limit error.
- Three workflows, note-autolike, ai-portraits-fragments and social-autolike, kept firing jobs through the outage, and unprocessed JSON files accumulated in one shared done/ directory.
- Measured at 05:20 the next morning, the files flagged needs_imagegen_thumbnail had gone from 43 to 57, with all fourteen additions left untouched.
- The instant Codex came back, five launchd lanes fired at once, each running a pickup script that reads the same directory.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Each repeat execution redoes the work for that item, so the price of the outage includes at least ten extra runs across eleven files once service returned.
- decision Schedule spacing is off the table as a remedy: the author's position is that after recovery every lane starts simultaneously however the intervals are tuned, which puts the fix in the claim step or nowhere.
- constraint A lock that can outlive its own process buys duplicate-safety at the cost of a stall path, so the easy shell version is not deployable until someone writes and owns the crash-recovery procedure.
- precedent With exclusion sitting in the pickup script, every new lane pointed at done/ has to implement it too, and one that skips it reintroduces the race for all the others.
launchd's contract is the clock. It runs run-codex-funnel.sh at the times listed in the plist and does not ask whether the previous run finished or whether another lane is already holding the same file [10]. The author of the dev.to post calls that correct by-design behavior [10] and wrote that "launchd is a scheduler, not a mutex" [11]. It is the one component in the incident that did exactly what its documentation promises. Mutual exclusion over a file queue belongs to the pickup script [12].
For months the offsets were enough. Each lane starts at a slightly different time and the target files arrive one at a time, so two processes rarely see the same JSON [13]. In com.lily.codex-note-funnel.plist, StartCalendarInterval lists 10:40 and 16:40 [9], which is two invocations a day for that lane, six hours apart [27].
Each pickup script lists done/ with ls or a glob and starts on what it finds [6]. The post's diagram shows lanes A and B and funnel-pm acquiring job_001 at the same moment, with 57 files backed up and every lane rushing the head of the queue [28]. Eight of the eleven files in the funnel-pm batch were processed two or more times [8], about 73 percent [23]. The floor for eleven items is seven files at two runs, one at four and three at one, so 21 executions, ten of them repeats [24].
The lock file most people reach for first has a race of its own. In the example script the test is `[ ! -f "$LOCKFILE" ]` and the claim is `touch`, and in the gap between them a second process can pass the same test, so both touch and both proceed [17]. If a process crashes with the lock in place, every later run sees the lock and exits immediately, processing stalls, and a separate crash-recovery procedure is needed, which the author counts as added operating cost [18].
flock closes the check-and-claim gap inside a single call. It does not help here, because launchd spawns a fresh process each time and file descriptors are not inherited, so it cannot exclude one lane from another [19]. The published excerpt ends mid-sentence at that point. The headline credits one line of mv with killing the four-times duplicate run [20], and the requirement that line has to meet follows from the two failures above: test and take in one operation [25], leaving behind no state that stalls the other lanes [18].
The author is direct about what kind of error this was, writing that "This isn't a probability problem; it's a structural one" [14]. The post lists the procedural reflexes it rejects, among them checking the logs more often and retrying by hand, and calls them bandages that leave the cause in place [26]. The system underneath is a side business the author says went from nothing to 1.2 million yen a month in six months, built with Claude Code [21].
What to watch
- Whether the mv-based claim step has a recovery path for a file claimed by a process that dies mid-run, since the lock-file version stalled every later run.
- Whether the other lanes, including note-autolike and social-autolike, get the same claim step; the duplicate counts published cover funnel-pm only.
- The next day-long usage_limit_exceeded outage, which is the condition that reproduces the simultaneous start.