Skip to content

Build1 publisher3 min readPublished

A wrapper's echo $$ put the shell's PID in the pidfile the deploy script trusted

For months a nightly job processed the same batch twice, always within fifteen minutes of a deployment. The author of a dev.to field note spent 48 hours suspecting the broker before checking which process the pidfile named.

The Engineer · Build desk

Illustration accompanying A wrapper's echo $$ put the shell's PID in the pidfile the deploy script trusted

What happened

  • Duplicate batches showed up only within fifteen minutes of a deployment and never on a quiet night, the one clue the author says he trusted.
  • On the host, pgrep -f worker.py returned two process IDs for the same consumer while the supervisor reported the service as stopped.
  • The deploy wrapper ran echo $$ > worker.pid before starting Python, so SIGTERM hit the shell and left the worker orphaned under init with its queue connection intact.
  • A three-file harness in the post reproduces the failure in about five seconds and records pid, ppid and pgid as JSON lines at every event.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint A pidfile written by a shell that does not exec cannot answer whether the consumer died, so a supervisor reading it is reporting the state of a wrapper.
  • decision Duplicate delivery now has a cheaper first move than auditing broker acknowledgement settings: resolve the pidfile's PID and compare it against the worker's PPID before touching queue config.
  • cost The broker hypothesis held for months, and the shutdown path went unexamined for the 48 hours of active investigation.
  • capability Testing a shutdown path no longer needs a production deploy window: three files and a kill command produce the orphan locally.

`echo $$` returns the PID of the shell that runs it. The deploy wrapper wrote that number into worker.pid and then started python3 as a child, so the recorded PID belonged to a shell holding no queue connection [7]. The harness published with the field note keeps the defect on purpose. Its second file is commented "deliberately does NOT exec, mimicking the deploy wrapper" [14]. Skip the exec and the shell stays alive as the worker's parent. The PPID comparison in the post's diagnostic is what exposes that gap [19].

From there the shutdown ran in order: the deploy script sent SIGTERM to the PID in the file, the supervisor reported the service stopped and started a fresh instance, and the old Python process kept running, reparented to init, still holding its queue connection [6]. Two consumers drained the same queue, and neither knew the other existed [1]. The nightly job usually takes eleven minutes, and the metrics showed the same batch processed twice [2].

The first theory was a missing handler. The author installed `signal.signal(signal.SIGTERM, handler)` and logged every delivery [8]. "That changed nothing, which should have been decisive evidence instead of a puzzle," the author wrote [9]. The second theory was a stuck network read, so the code got timeouts everywhere and eventually a forced `os._exit`, and there were still two processes [10]. Both fixes went into the Python worker, the one process that was never sent a signal [21].

The check that resolves it is one line: `ps -o pid,ppid,pgid,sid,cmd -p "$(cat worker.pid)"` [11]. If the PID in the file resolves to an `sh` process and the Python PID has a different PPID, the answer is in front of you [11]. The same output carries the process group and session IDs, and the post says those will tell you whether `kill -TERM -PGID` would be a safe broadcast or a small disaster [12].

The harness makes the evidence binary. A start record with no signal record means the handler never ran, and the post calls that missing line "the difference between a logging problem and a supervision problem" [15]. On Linux you can also read the disposition directly: `/proc/<pid>/status` publishes SigCgt, SigBlk and SigIgn bitmaps, and the post's sigset.py decodes them into signal names [16]. Run it against both processes and "you will usually see SIGTERM in SigIgn or absent from SigCgt on one of them", the author wrote [17]. The post also flags PID 1, where the kernel does not apply default terminating actions [18].

This is one host and one deploy script, and the author describes the write-up as "a field note, not a tutorial about signal theory" [20]. Three conditions have to hold for it to be your bug: a pidfile written by a shell that does not exec, a deploy script that sends SIGTERM to whatever that file says, and a supervisor whose liveness answer comes from the same PID [6][7].

What to watch

  • The post does not report a fix landing in the deploy script; a follow-up showing worker.pid resolving to python3 after an exec would close it.
  • Whether supervisors that kill by process group or cgroup instead of by pidfile PID leave the same orphaned consumer behind.
  • Anyone running the five-second harness in a container where the wrapper is PID 1, given the post's note on default terminating actions.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories