A 28-host Debian 12 cutover, and the 02:13 failure Ansible could not have prevented

Nothing in the play caused the stop at 02:13. A Debian 11 host in the payments group refused the old OpenSSH cipher list ^[2], which is a property of sshd and the client config on that pair of machines, not of any task in the run. Work backwards from the 41 quiet minutes and the run began around 01:32 ^[16]. The write-up never says what moved first, and that gap is the interesting part: during a distribution cutover, the transport you automate over is one of the things being upgraded.

The rerun tally is the number worth keeping. Twenty-three hosts ok, four changed, one failed ^[7] comes to 28 ^[15], which is the whole fleet ^[1]. The second pass therefore reached every host including the one that had dropped out, and the only failure left was a stale internal repository key ^[7]: content, in the play, fixable by whoever owns the key. The unrecoverable class of failure was the transport, and the transport is the part ansible-core 2.17.7 in a pinned virtualenv ^[3] has no authority over.

The month before the cutover says the same thing. Four separate ways to lose a connection before a single task runs: a stale known_hosts entry, a rotated host key, an expired jump-host certificate, an orphaned control socket under /tmp ^[8]. Call it one new class of transport failure per week ^[17], with no distro upgrade to blame. That is the sense in which "it's just SSH" is wrong and still worth respecting. Inventory groups, facts, handlers, check mode and a run history you can query afterwards are all real ^[5], and the APT source went in through deb822_repository with nginx reloaded only when the file changed ^[12], which no shell loop gives you across hosts that differ as much as gunicorn nodes, workers with their own systemd limits, and reporting boxes holding an NFS mount ^[6]. None of it starts without a working socket ^[4].

The Celery incident belongs to a different family. Six workers restarted on a run that should have reported no changes, because a task was written with state: restarted ^[11]. The module was idempotent; the play author was not. That is a code review problem rather than a tool problem, and it is precisely what the "safe to rerun" belief conceals ^[9]. The unit change underneath it, from Restart=always to Restart=on-failure after a wrong RabbitMQ credential produced 9 GB of logs in one on-call shift ^[10], moves the failure mode from loud to quiet: the next bad credential leaves stopped workers instead of a log flood, and queue depth has to do the detecting.

One more thing about the source itself. The published example breaks off at "- name: Install worker" without the corrected task ^[13]. On a run whose expensive lesson was a transport failure and whose self-inflicted one was a shortcut inside a handler, that missing snippet is the part everyone else would have copied.