Green is four claims wearing one badge, and nobody instruments the fourth

An operator running about 283 scheduled shell jobs across a handful of machines has published an account of five incidents in which every health signal they had was correct and the job still was not doing its work ^[1][3]. The useful part is not the bugs, it is the taxonomy: for a scheduled job, "green" is a conjunction of four independent claims wearing one badge ^[5].

The four are: the test passes, the test asserts the thing the job actually does, the job is really scheduled, and it is scheduled on the box where its consumer exists ^[5]. In this fleet each job declares its own schedule in a header comment, ships its own `--test`, and gets wired into cron automatically once that test passes ^[2]. That covers claim one mechanically. Claim two was covered by discipline rather than tooling. For claim four, according to the author, there was nothing at all, and claim four is the one that runs silently for weeks ^[6].

The worked example is a node health job that fuses four inputs into one label, writes it to a state file, and with `--edge` prints a line only when the label changes ^[7]. Cron runs it every five minutes and a separate log records transitions ^[8]. The `--edge` path wrote the new state and then read the previous value back out of the file it had just written, so the previous value equalled the current label by construction, the equality test held on every run, and the transition log could not append ^[9]. This was invisible for six weeks in production ^[10].

Every liveness signal said fine, and each was individually correct: the cron entry existed, the process ran on schedule, exit code was 0, and the state file mtime stayed current because the job touches it on every successful evaluation by design, so that a long stable value does not read as a dead job ^[11]. At twelve runs an hour over six weeks, that is roughly 12,000 clean exits with an inert detector ^[2]. The only observable was a log that had stopped growing, and a quiet transition log looks exactly like a machine that is behaving ^[12].

The scheduling detail underneath is the part worth stealing. The health job runs at `2-59/5`; a vitality roll-up runs at `2-59/15` and force-refreshes the same label through the same tool, so at minutes 2, 17, 32 and 47 two writers of one state file fire in the same second, four times an hour by schedule rather than by luck ^[14]. Without a lock, a reader that catches the file truncated gets an empty previous value, which never equals a real label, which is a spurious edge ^[15]. Those spurious edges were the only lines the log ever produced: on the live file the author counted 33 lines naming both ends of a transition and 4 that do not, the four being pre-fix survivors ^[16][17]. That is 37 lines total, about 11 percent of them unattributable ^[1].

Two things follow. A log line that names one end and no date cannot tell you what changed, so even the accidental firings were unusable after the fact ^[18]. And moving the read one line nearer the write instead of before it would have fixed the equality test while leaving the race intact, producing a log that grows again, which is the signal they were missing, which is how an unfixed bug gets closed ^[19].

What to watch: whether placement gets an assertion rather than a habit. The same fix immediately tripped the deployed `--test`, which runs the tool in a sandboxed HOME and asserts that no live sense file moved, and nothing was actually wrong ^[20]. Note also the author's own caveat: the numbers were re-measured on one 16-core Ubuntu 24.04 box, two came out different from the fixing commit, and one mechanism did not reproduce at all ^[4].