A crash between the claim and the work makes the copied webhook dedupe swallow events

Two writes, and nothing binds them. The claim goes into `processed_events` first; the work commits second ^[2]. Anything that kills the process in between leaves a key that says handled next to a side effect that never ran: a pod restart mid-deploy, an OOM kill, a dropped connection to the downstream API, a SIGTERM the worker does not handle gracefully ^[4].

The order of those two writes picks which failure you get. Work first, crash before the claim, and the next delivery runs the side effect twice; the post calls that loud and recoverable ^[7]. Claim first, crash before the work, and every later delivery takes the conflict branch and returns 200 ^[3]. There is no error, no retry and no alert ^[5]. "The event is just gone, and you find out when a customer emails about the thing they paid for," the post's author wrote ^[6].

The deliveries keep arriving because at-least-once delivery is working as designed: the handler commits, the response is lost or lands after the sender's read timeout, and the sender sends again ^[8]. Stripe retries failed events for up to three days in live mode, and anything can be resent from the dashboard on top of that ^[9]. GitHub allows redelivery of anything from the last three days out of its UI ^[10]. The window stays open whatever timeout you set ^[11].

When the side effect is a write to the same database, the fix is one transaction around both statements. A crash rolls back the claim along with the work, and the next retry claims the key cleanly ^[12]. "ON CONFLICT still does the concurrency half, and the transaction does the crash half," the author wrote ^[13].

That fix has a price in throughput. The transaction stays open for as long as the work runs, so a third-party call with a thirty second timeout holds a database connection and a row lock for thirty seconds per event ^[14]. Read a few hundred events a minute as 300: five a second, each holding for up to thirty seconds, is 150 connections in flight ^[16]. The post says you run out of connections at that volume, and that at low volume nobody notices ^[15]. The 150 only applies if the call runs to its timeout. If it returns in a second, the same rate holds five ^[17].

Emails, card charges and other people's APIs do not roll back with your transaction ^[18]. For those the post's table adds `status` ('in_progress' or 'done'), `claimed_at` defaulting to `now()`, and a nullable `completed_at` ^[19]. The claim becomes an upsert that sets status back to in_progress and refreshes claimed_at, taking the key off a worker that has clearly died ^[20]. On retry the handler asks whether the key is done, and a bare row no longer answers that ^[22]. In the source text the upsert breaks off at the WHERE keyword, so the condition that decides when a key can be taken back is not shown ^[21].

That predicate is where the design work sits. Too short an interval and a slow but living worker has its key stolen; too long and a lost event outlives the three-day replay window the providers give you ^[23]. In my view the extra column is worth the migration, because reclaiming an in_progress key hands the work back to the retry, and a duplicate leaves a trace ^[20].