Build1 publisher2 min readPublished
A workflow_run listener reads the shutdown-signal line before it retries a dead CI job
On a self-hosted spot runner, an instance AWS reclaimed and a genuinely broken test both produce the same red X, so this pipeline reads the raw job log for four host-only strings and re-runs only the jobs the host killed.
The Engineer · Build desk
What happened
- A backend deploy went red on Tests (shard 1/2) when AWS took the self-hosted ARM64 spot host back mid-run, and GitHub renders that as the same red X a failing test produces.
- The second test shard died 13 seconds later on the same host, and the frontend deploy then aborted on purpose, because it will not publish without a successful backend deploy for that commit.
- The obvious triage command, gh run view with --log-failed, prints nothing at all for this class of failure.
- A listener now reads the failed jobs' logs and matches four strings only the runner host can write, then calls rerun-failed-jobs only on a match, leaving unmatched runs red.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The repair job spends GitHub-hosted minutes on every failed or cancelled run of the three listed workflows, because a self-hosted copy would sit in the queue behind the dead pool it is meant to fix.
- decision The attempt-3 ceiling converts a repeating preemption into a person's Auto Scaling Group ticket instead of an unbounded retry loop.
- constraint Because workflow_run only fires from the default branch copy, nobody can watch this work on a pull request; the pre-merge check is a hand-run of the script against a known run id.
- capability Lifting the Auto Scaling Group ceiling is what lets Capacity Rebalance launch a replacement before the old box goes. That replacement is the condition under which a retry finds a host at all.
Because the command prints nothing, it reads as a broken tool and the log that holds the evidence stays closed [7]. The command that works names the job: `gh api repos/myorg/myapp/actions/jobs/<job-id>/logs | tail -20` [8].
The tail is legible once you have it. Progress dots reach 75 percent at 19:31:20Z, `##[error]The runner has received a shutdown signal` lands at 19:31:31Z, and `##[error]The operation was canceled` at 19:31:44Z [2]. That is 11 seconds from the last live output to the signal, and 24 seconds from progress to dead [1][2]. pytest is logged in that final second as an orphan process, pid 18652 [4]. No test had failed [3].
The listener wakes on the run conclusion and decides on the log text. It fires on any completed run of CI, Deploy Backend or Deploy Frontend, and the job body runs only when the conclusion is failure or cancelled [9]. The marker list covers more than preemption: one of the four strings is "lost communication with the server" [10]. A match calls rerun-failed-jobs with a token whose only scopes are `actions: write` and `contents: read` [14].
Two other refusals sit alongside the attempt ceiling. A log that could not be read, because it expired or is still uploading, is not evidence of anything [12]. And if a newer run of the same workflow and branch exists, re-running an older attempt only fights the concurrency group [12].
Whether a re-run finds anything depends on a runner coming back [18]. Capacity Rebalance subscribes the Auto Scaling Group to the two-minute spot interruption notice so a replacement launches before the instance is reclaimed [19]. The CDK had `capacityRebalance: true` a few lines above `maxCapacity: 1` [20]. An Auto Scaling Group cannot exceed MaxSize, so there was nowhere to put the replacement and the feature degraded to terminate-first. That degradation is the outage the feature exists to prevent [21]. With minCapacity, maxCapacity and desiredCapacity all at 1, one host is the entire pool [20].
Moving the ceiling to 2 while desiredCapacity stays at 1 puts a second instance online only during the handover, and the notice window bounds that overlap at roughly two minutes of double capacity per interruption [22][3].
That change has a precondition. Each instance registered its runners under fixed names with `--replace`, as `./config.sh --name myapp-runner-$i` [23]. During the handover two hosts are alive at once, and both want that name. I would want per-host names in place before the ceiling moves.
What to watch
- Whether any job's own output can emit one of the four marker strings, which would re-run a genuine failure without anyone noticing.
- Whether the frontend deploy's refusal-to-publish gate clears automatically once the re-run of the backend deploy passes.
- Whether the two-file script stays dependency-free on the gh CLI as more workflows are added to the trigger list.