Published · 6d agoBuild3 min read
The outbox relay's missing integer: how one unsendable row stalls the whole pipeline
A developer's write-up of running the outbox pattern in production finds the textbook loop breaks on error to preserve ordering, and never counts attempts. That combination jams forever.
Written for builders.See today for builders

What happened
- The outbox pattern as commonly described is: write the event into the same transaction as the row, poll the table, send to Kafka, mark it sent.
- The author built an outbox relay, ran it between two services, and hit three separate ways for the 'obviously correct' pattern to go wrong in practice.
- The relay's job is to claim a batch of unsent rows, send each one, and mark it SENT on a broker ack.
- The first version of the relay loop caught the exception, logged a warning that the row stays NEW and will be retried, and then called break.
- Using break instead of continue is deliberate and correct on its own, because per-partition ordering must be preserved: if row 3 fails, rows 4 through 20 should not jump ahead of it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
A developer publishing on dev.to ran an outbox relay between two services and wrote up the failures, and the first one is a two-line loop that every version of the pattern online implies but nobody prints [2]. The pattern as usually stated is four sentences: write the event in the same transaction as the row, poll the table, send to Kafka, mark it sent [1]. The relay that implements it claims a batch of unsent rows, sends each one, and marks it SENT on a broker ack [3]. The first version of the loop caught exceptions, logged that the row stays NEW and will be retried, and then called `break` [4]. That `break` is not sloppiness. It is there so that per-partition ordering survives a failure: if row 3 cannot be sent, rows 4 through 20 must not overtake it [5]. On its own the choice is correct. What is missing is an integer. Nothing counts how many times row 3 has already failed [6]. If the cause is permanent rather than transient, which the author lists as a broker that is genuinely down for that topic or a payload the producer cannot serialize, the relay retries that same row on every poll forever, and every row behind it accumulates behind a jam that will not clear by itself [7]. There is no ceiling and no signal; the only symptom is a NEW count in the table that keeps rising [8]. Ordering preservation and unbounded retry compose into an unbounded backlog: the block never ends, and every subsequent insert for that partition joins it [23]. The repair is an attempts column read, incremented, and compared against a maximum [9]. Past the limit, which defaults to 10, the row is written to status FAILED with its attempt count [10][11]. The important detail is that the give-up path uses `continue`, not `break`, so the jam the dead row was causing clears while the rows behind it, which may still succeed, keep trying [12]. Rows that have not exhausted their budget still get `break` after their counter is persisted [13]. Bounding retries this way also bounds the blast radius: a permanently unsendable row costs at most 10 failed sends before it is parked, instead of one per poll indefinitely [24]. The other half of the value is operational. A FAILED row is something a human can query, alert on, and replay by hand once the root cause is fixed; an infinitely retried NEW row surfaces nowhere [14]. That is the difference between a pipeline with a queue depth you can chart and one you have to notice. The uncomfortable part, in the author's telling, is that the counterless version passed every test that existed. The suite covered the happy path and a single transient failure that resolved itself [15]. Nothing covered "this row will never succeed", because writing that test requires first accepting that a message can be permanently unsendable, which is exactly the assumption the four-sentence version of the pattern encourages you to skip [16]. The same write-up flags a related gap: HTTP clients propagate trace context automatically, so a call from order into inventory carries the caller's trace [17], while a polled outbox row does not, because the originating request is long finished by the time the relay reads it and Kafka has no concept of a trace [18]. Left alone, each consumed event starts its own trace and the inventory hold appears in Jaeger as an orphan [19]. The author's fix is manual: a trace_id column written from MDC at insert time [20], re-attached by the relay as a Kafka header next to the dedup message id [21], and restored by the consumer before processing so logs and spans stay tagged [22].
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The outbox pattern as commonly described is: write the event into the same transaction as the row, poll the table, send to Kafka, mark it sent.
ReportedView cited source - [2]
The author built an outbox relay, ran it between two services, and hit three separate ways for the 'obviously correct' pattern to go wrong in practice.
- [3]
The relay's job is to claim a batch of unsent rows, send each one, and mark it SENT on a broker ack.
ReportedView cited source - [4]
The first version of the relay loop caught the exception, logged a warning that the row stays NEW and will be retried, and then called break.
ReportedView cited source - [5]
Using break instead of continue is deliberate and correct on its own, because per-partition ordering must be preserved: if row 3 fails, rows 4 through 20 should not jump ahead of it.
ReportedView cited source - [6]
The bug is what is missing: nothing counts how many times a failing row has failed.
ReportedView cited source
Sources & coverage · 1 publisher
The reporting this story was synthesized from, earliest first. Every link goes to the original.
- dev.toQihu Zhang6d agoThe outbox pattern is four sentences in a blog post. Here are three incidents from running it.
Cited in this coverage: dev.to post by danzizhangdev
- dev.toRamesh Yara5d agoThe Outbox Pattern Is Not Enough

