Skip to content

Build1 publisher2 min readPublished

A pod restart between charging and committing the offset bills the customer twice

Kafka promises at-least-once delivery, so a consumer that dies after the payment call but before the commit replays the event on restart. A dev.to tutorial puts the guard in a database primary key, keyed on an event ID the producer sets.

The Engineer · Build desk

Illustration accompanying A pod restart between charging and committing the offset bills the customer twice

What happened

  • Kafka's delivery guarantee is at-least-once, not exactly once, so a consumer must expect to be handed the same record more than once.
  • The exposure sits between processing and the offset commit: if the process dies after the work and before the commit, the restarted consumer reads from the last committed offset and gets the event again.
  • That window opens on routine operations, since every deploy, every scale-up and every Kubernetes pod restart triggers a consumer group rebalance.
  • In the tutorial's naive listener, the redelivered event calls paymentService.charge a second time with no error and no stack trace, producing a double-charged customer.
  • The proposed remedy is a processed_events table with event_id as the primary key, so the database rejects a second insert for an event the consumer has already handled.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Deduplication buys correctness with a database write on the path of every message, and the table accumulates one row per event until someone writes a retention job.
  • exposure Until the guard exists, the detection channel for this class of bug is the customer, since a duplicate charge raises no exception for logs or alerting to catch.
  • decision Teams adopting the pattern have to decide where the atomic boundary goes, and a side effect outside the database cannot join the insert's transaction.

The guard is a uniqueness constraint. The tutorial's table is one column plus a timestamp: `event_id VARCHAR(255)` as the primary key, `processed_at TIMESTAMP NOT NULL DEFAULT now()`, and the database itself refuses to store the same event_id twice [9]. No branch in the listener has to be correct for that to hold. Where the ID comes from is the part that has to be right: it must be a stable ID set by the producer and carried in the payload, an eventId or an orderId, because a `UUID.randomUUID()` minted at consume time makes every redelivery look new and the whole scheme pointless [8]. The tutorial maps the row to a JPA `@Entity` with `@Id private String eventId` [10].

The failure being replaced is quiet. On redelivery `paymentService.charge(...)` runs a second time with no error and no stack trace, and what surfaces is a customer with two charges and a support ticket [5]. "The bug isn't in Kafka. It's an assumption that your consumer would ever see each event only once," the tutorial says [6]. There is nothing to catch, so the fix is not a try/catch; the operation has to be safe to run any number of times with the same result as running it once [7]. The tutorial's picture for the current state of `charge()` is a light switch, where every press changes the state, against an elevator button pressed five times for one elevator [11].

Whether the pattern transfers turns on atomicity. The tutorial states that two things must happen together or not at all, recording the event and performing the side effect, and the supplied text breaks off at that sentence [12]. If the work is a row in the same database as `processed_events`, one transaction covers both and the primary key is the whole guarantee. If the work is an HTTP call to a payment provider, the insert and the charge cannot share a transaction. Insert first, and a crash in between loses a charge. Charge first, and the customer is exposed to exactly the double charge the table was built to prevent. In that shape I would want the provider to accept an idempotency key of its own, with `processed_events` as a fast path only.

Every remedy in the excerpt sits on the consumer side: a producer-set event ID and a uniqueness constraint [13]. For a billing consumer that is the scope I would pick too, since the consumer is the component that can see the duplicate and owns the side effect. What the tutorial names as the trigger is ordinary operational traffic: a deploy, a scale-up, a pod restart [4].

What to watch

  • Whether the rest of the tutorial commits the processed_events insert and the side effect in one transaction, and against which database.
  • A comparison of this consumer-side pattern with Kafka transactions and read_committed isolation, which the excerpt does not cover.
  • A retention or partitioning step for processed_events once the table is carrying a row for every event a topic has ever delivered.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories