A service name missing from one shell variable kept the scheduler two weeks behind

The whole bug fits in the distance between a repository and a process. The deploy script asked the server for its commit SHA and got a correct answer, and it asked whether all containers were running and they were ^[8]. Both statements were true, but neither addressed the thing that had just changed ^[8]. By the author's own reading, the SHA proves the server pulled and says nothing about whether any particular container was recreated from the new image ^[17].

The service list is where it went wrong. A backend deploy rebuilt `SERVICES="$SERVICES backend celery-worker"`, and celery-beat is not in that string ^[5]. So the two containers named came up a minute before the checks ran, while the container holding the schedule had been up for two weeks ^[6]. Of the five green lines, none reads the beat schedule; the only signal that disagreed came from logging into the scheduler and asking it what it had scheduled ^[5].

The remedy worth reading is the age assertion. The verify step now loops over the services it asked to rebuild, reads `.State.StartedAt` with `docker inspect`, and dies unless the age is under 900 seconds ^[16], the fifteen minutes described in the post ^[1]. That is cheap and I would ship it. It also inherits the original blind spot, because the loop iterates `$SERVICES` and the failure was celery-beat's absence from `$SERVICES`, so a service left off that list is a service the loop never checks ^[2].

It does close the earlier instance. Annual billing shipped dead because the compose file passes environment variables explicitly, one line per variable, and the containers had started before the new variable existed ^[10]. Backend is in the rebuild list ^[5], so an age assertion would have failed that deploy on a backend container that was never recreated ^[3]. The author found it instead by curling the config endpoint and reading the value back, which the script had not done because it had no idea what value to look for ^[11].

That missing declaration is the class of bug. There is nowhere in the script where the deploy declares what it was supposed to make true, which is how a forum reply framed it: the gap is between a check that verifies state and one that verifies intent ^[13]. An age check is still state. It is a better proxy than a SHA because it is derived from the list of things this deploy claimed to touch, but a proxy is only ever as complete as that list.

The job being deleted had spent its existence logging "Cleanup completed: removed 0 expired results" and returning a completed status without deleting anything ^[2], which at one run an hour is twenty-four clean-looking successes a day ^[4]. Removing it with a pipeline that also reported success while doing nothing at least keeps the failure modes consistent.

The recurrence is the useful evidence. The author diagnosed the pattern, wrote it up, had it named by someone with no stake in it, and shipped the same class of bug again a week later, where a manual check caught it ^[13][14]. For the fifteen-minute assertion to transfer to another stack, the rebuild list has to be generated from what changed rather than typed once and edited by hand, because the check can only interrogate services someone remembered to name ^[2].