Build1 publisher2 min readPublished
Postgres keeps the outbox payload while a memory queue carries only the event ID
An experiment written up on dev.to publishes outbox events from an in-memory queue of committed event IDs and keeps polling only as a recovery scan, so that scan's interval now sets worst-case delivery latency.
The Engineer · Build desk
What happened
- The transactional outbox writes business data and its event in one commit, and the usual way to get that event to Kafka is a worker polling the table, which trades database load against delivery latency.
- In this implementation the commit pushes only the event ID into an in-memory queue, and a batch publisher loads the matching rows from PostgreSQL in batches before sending them on.
- Polling survives as a recovery worker that scans for events the fast path did not finish and returns their IDs to the same memory queue, so both entry points share one publishing pipeline.
- If the process dies after the commit, the queue contents go with it while the event stays in PostgreSQL, where the recovery worker finds it after restart and re-queues its ID.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The recovery scan interval becomes a latency budget for the failure case, so it has to be chosen against a delivery target instead of set to whatever keeps database load comfortable.
- cost PostgreSQL still answers the publisher on every batch, so what this design saves is scan frequency: the discovery query. A team whose pain was fetch volume gets little back.
- decision Teams choosing between Debezium and hand-tuned polling get a third option that lives inside the application process, with no Kafka Connect to operate and delivery timing owned by their own code.
In this design a queue entry is a pointer. The row in Postgres holds the payload, and memory holds an ID and nothing else [4][11]. So the queue is safe to lose. Every entry can be recomputed by asking Postgres which committed events were never published, the same question the polling design asked once a second [6][2].
The price is paid in the worst case, and the post states it directly. Normal latency is queue wait plus batch publishing; when the fast path fails, the recovery interval is added in front of both [8]. Take the post's own example settings, a one-second baseline poll against a ten-second recovery scan: an event that needs recovery waits up to ten times longer just to be discovered [9][14]. The post is right to say both happy-path latency and recovery latency should be measured [9].
The alternatives were considered first. Postgres LISTEN/NOTIFY and Debezium CDC are both named, along with the standard ways of making polling cheaper: batching, indexes, SKIP LOCKED, partitioning, longer intervals [12]. On the CDC route the author is precise about what it costs to run. "For many systems, Debezium is probably the right choice. But it also adds Kafka Connect, Debezium, and more infrastructure to operate," the post says [13]. The stated goal was to test a simpler application-level approach [12].
The polling tradeoff did not disappear from this design. It now applies to the recovery path, which is the path that runs after something has already failed [6][8]. The post does not describe deduplication for an event the fast path published without recording that it had [15]. A publish to Kafka and a status write in Postgres are two operations, and the recovery worker scans for events not successfully processed by the fast path [6]. If the publish succeeds and the status write does not, that scan would return work that has already been delivered.
What to watch
- Whether the author follows up with measured happy-path and recovery latency from a real workload, since the write-up gives formulas and not figures.
- Whether an idempotency design appears for events the recovery worker re-queues after a partial publish.
- A comparison against PostgreSQL LISTEN/NOTIFY, which the post names as an alternative but does not test.