Skip to content

Build1 publisher3 min readPublished

Three services you can delete: queue, cache and search in one Postgres

A side project swapped RabbitMQ, Redis and Elasticsearch for SKIP LOCKED, an unlogged table and a GIN index. The queue arithmetic shows where that choice stops working.

The Engineer · Build desk

Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction

What happened

  • A developer building AI agent infrastructure for agents that research, draft and publish content needed three things beyond the main database: a job queue, a cache for repeated lookups, and search over article text.
  • His first instinct was RabbitMQ or Redis Streams for the queue, Redis for the cache, and Elasticsearch or Meilisearch for search.
  • He counted the services: one Spring Boot app plus three more moving parts, each with its own Docker image, its own failure modes and its own 3 AM pager behavior.
  • He instead did all three jobs in Postgres: a table with SKIP LOCKED for the queue, an unlogged table for the cache, and a tsvector column with a GIN index for search.
  • The resulting system is one schema, one backup strategy and one connection pool.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

A developer running a handful of content agents needed three things beyond his main database: a job queue, a cache for repeated lookups, and search over article text [1]. Instead of adding services he put all three in Postgres, and that matters because at small scale the marginal service is not a feature but a Docker image, a distinct failure mode and its own 3 AM pager behaviour [4][3].

The default stack he reached for first was RabbitMQ or Redis Streams for the queue, Redis for the cache, and Elasticsearch or Meilisearch for search [2]. Counting processes changed the decision: one Spring Boot app plus three more moving parts is four things to run where one would do [3][24]. What replaced them was a table claimed with SKIP LOCKED, an unlogged table for the cache, and a tsvector column with a GIN index for search, all inside one schema with one backup strategy and one connection pool [4][5].

The queue is the part he works through in code, and it is the least exotic. The naive approach, SELECT followed by UPDATE WHERE status = 'pending', races between workers [7]. Postgres has shipped the fix since 9.5: SELECT ... FOR UPDATE SKIP LOCKED, which skips rows another worker already holds [6]. The claim is one statement, an UPDATE whose subquery does FOR UPDATE SKIP LOCKED with a LIMIT and a RETURNING * on the outside, so concurrent workers get disjoint batches without a second round trip [8]. No advisory locks, no leader election, no broker [9]. The docs cover it under row-level locking, and Crunchy Data has a deep dive on native Postgres queuing [20]. Jobs survive restarts because they live in a table that is already in the backup [11].

Now the arithmetic, because this is where honest limits live. The poller in the post is a scheduled method firing every 2000 ms and claiming 5 jobs [10]. That is 150 jobs per minute per worker [21], with a worst case wait of 2 seconds before a ready job is picked up and about 1 second on average [23]. The author's own stated breaking point is millions of jobs per minute, streaming semantics, or long retention for event replay, at which point you want a real broker [14]. Reaching one million jobs per minute at that batch and interval would take roughly 6,700 workers [22], which is another way of saying the gap between this pattern and Kafka is several orders of magnitude wide, not a close call. What you are accepting in exchange is at-least-once delivery with polling latency rather than push, and no replayable ordered log [13]. The thing you skip is bootstrapping a broker, writing consumers, configuring dead-letter exchanges and understanding prefetch [12].

The precedent cited is real but uneven: Bauer's essay "PostgreSQL for Everything" was on the Hacker News front page this week, Contentful rebuilt full-text search on Postgres rather than a separate cluster, Instacart built search infrastructure on Postgres, and the Guardian moved parts of its platform off MongoDB onto Postgres [15][16][17][18]. The author is explicit that he is one developer in Dhaka with a VPS and has run these patterns in his own projects, not at giant scale [19].

Watch your own arrival rate against the poll interval. If depth grows between polls, tune batch and interval first; that is a config change, not a new pager rotation. And note that the queue pattern here comes with a stated failure boundary [13][14] while the unlogged cache table and the GIN search index are named rather than stress-tested [4], so treat those two as the parts you still have to prove.

Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories