Build1 publisher2 min readPublished
Changing the operation key on a retry left the lab with a second job
A small SQLite harness kills its worker before and after COMMIT, and whether the retry reuses the recorded operation key decides between replaying the job that already exists and enqueueing another one.
The Engineer · Build desk
What happened
- The lab starts from an agent call to create_report_job("weekly") that returns no response, leaving the runner to decide whether to try again.
- If the database committed and only the acknowledgment was lost, a second create call enqueues a second job; if the worker stopped before commit, doing nothing leaves no job at all.
- The worker writes the job row and a receipt row in one transaction, the receipt tying an operation key to the requested report and the job ID it produced.
- On another attempt the same key and report return the recorded job ID, the same key with a different report is rejected, and a new key counts as a separate intended operation.
- The forced exits kill a subprocess before or after commit to stand in for a worker disappearing without an acknowledgment, and the post says they are not a network-timeout or power-loss test.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Because a timeout alone cannot say whether the write happened, the retry decision cannot be derived from the error at all; it has to be answered from state the service already recorded.
- precedent The post is explicit that the guarantees here are the example's own and not promises made by an arbitrary tool API, so a tool that wants to be safe to retry has to publish a key parameter and enforce it.
- decision Whoever writes the tool now has to decide where the key comes from, because deriving it from the payload merges a genuinely deliberate second weekly report with a retry of the first.
- capability Both failure modes can be reproduced on a laptop with the standard library and no credentials, before anyone commits to an API shape.
Enforcement lives in the worker's first two statements. Each attempt opens with BEGIN IMMEDIATE, taking the write lock before it looks up the receipt for the key [13]. The receipts table declares key as PRIMARY KEY and job_id as NOT NULL UNIQUE [14]. One receipt per key, one job per receipt.
When the key is already recorded against a different report, create_job raises ValueError("key_payload_conflict") [15]. The case table expects that second call to exit 20 with the job count still at one [20]. The finally block closes the connection, and the code comment notes that this also rolls back the conflicting request's open transaction [17].
The faults are blunt. They are os._exit(17) before COMMIT and os._exit(18) after it, carrying a comment that there is no Python cleanup and the process ends without an ack [16]. The harness gives each subprocess ten seconds and does not raise on a nonzero exit [25]. A worker that shuts down politely is not the worker you are worried about.
Two rows carry the retry decision, and the post notes that both start from a committed job whose caller received no acknowledgment [29]. Retrying under op-a leaves one job and exits 0 [18]. Retrying under op-b leaves two [19]. The only difference between those second calls is the key [28].
Then there is the fifth row, which runs with no fault at all and a second call under op-b, and also ends with two jobs and exit 0 [22]. Its recorded state is identical to the case-3 duplicate: two rows, clean exit [26]. After the fact, the database has no way to separate them. The post puts it plainly: identical parameters do not necessarily mean duplicate intent, someone may genuinely request two weekly reports, and using only a hash of the payload as the operation key would conflate those requests [23].
For any of this to transfer, the job and the receipt have to commit together [6]. In the lab they are rows in one temporary SQLite file, with no HTTP call and no model anywhere in the path [11]. Where the queue is one service and the receipt store is another, that single transaction is the part you have to rebuild.
An agent runner cannot add this guarantee from outside. The post's recommendation is to preserve the identity of the intended operation instead of inventing a new identity for each attempt, and to use a service that actually enforces that identity [5].
What to watch
- Whether the tool APIs an agent actually calls accept an operation key, and what they document doing when the same key arrives with a different payload.
- Whether the runner mints one key per intended operation or one key per attempt.
- Results from the same harness when the parent process dies mid-call, not only the worker.