Skip to content

Build1 publisher3 min readPublished

A primary key and a lease column let Kestrel run durable workflows inside Postgres

Kestrel Workflows built durable execution out of Postgres row locks, constraints and leases, and published the schema that does it. The design keeps a second stateful system out of the path of every incident response.

The Engineer · Build desk

Photograph accompanying A primary key and a lease column let Kestrel run durable workflows inside Postgres
Photo: usekestrel.ai

What happened

  • Kestrel Workflows runs durable execution with no orchestrator process: each application server embeds a workflow library and talks straight to Postgres.
  • Crash recovery is a lease and a sweeper: workers heartbeat the rows they own, and a periodic query re-enqueues any execution whose lease has expired.
  • The team almost used Temporal or AWS Step Functions and reconsidered after looking at what a second stateful system in the critical path would cost.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Postgres now holds the system of record and every in-flight execution, so a single database incident stops the workflows that would respond to it.
  • decision Adopting this puts durable execution in front of whoever owns the database schema, because the queue, the leases and the checkpoints are tables in it.
  • cost The recovery delay is paid by whatever waits on the workflow: a dead worker's row cannot be reclaimed until its lease lapses and the sweeper next runs.

Idempotency here is a database constraint. Each step's checkpoint row has a primary key, so a rerun after a crash collides on insert and reads the prior result instead of repeating the side effect [4]. Claiming works the same way: the queue table is read with SELECT ... FOR UPDATE SKIP LOCKED, which the InfoQ article says hands each row to exactly one worker with no message broker, no leader election and no external lock service [3].

Constraints don't cover liveness. Workers heartbeat the rows they own, and a periodic query re-enqueues any execution whose lease has expired [5]. A dead worker's row therefore stays unavailable until its lease lapses, so recovery delay is the lease duration plus the sweep interval. The article does not state either figure, or a claim rate [16].

The schema carries that bookkeeping. Of the six columns named in the published workflow_executions DDL, two exist for the lease: owner_id, commented "which worker holds the lease", and lease_expires, "when the lease lapses" [2][1]. Rows arrive in that table from every trigger, PagerDuty alerts and GitHub webhooks included [2].

The author does not dress up the trade. Storing workflow state in the primary database "makes observability a plain SQL query and collapses reliability and security to one dependency" [6]. Both halves arrive together. The database holding the system of record now holds every in-flight execution, which also means the workflow engine is something a person with write access can UPDATE by hand.

Temporal and AWS Step Functions were the usual advice, and the team nearly took it [13]. What stopped them: another stateful system to deploy, secure, monitor and upgrade, sitting in the critical path of every workflow [7]; step payloads that for Kestrel include infrastructure topology, source code and logs going to a system with its own access controls and audit handling [8]; and a key-value data model optimised for workflow state, which the author says makes ad-hoc analysis less straightforward than querying Postgres directly [9].

That reasoning transfers only if two conditions hold. You already run Postgres as a system of record you trust under load, as the team says it does [15]. And your step rate has to leave headroom on that instance for claim queries and heartbeats. Kestrel's example workflow triggers on a failing Kubernetes workload, runs an AI root cause analysis, generates a fix, waits for a human to approve it in Slack, then opens a GitOps pull request [10]. That step rate is bounded by people. A pipeline firing thousands of short steps a second puts a different load on the same tables.

In a shop that already runs Postgres well, I would take this trade. The failure it is built for is ordinary: a system rescheduled mid-workflow, which in Kubernetes, the team writes, "it most likely will" be [11].

What to watch

  • Publication of lease duration, sweeper interval and claim throughput, the numbers needed to compare this design against Temporal under load.
  • Whether claim queries and heartbeats start contending with system-of-record writes as workflow volume rises, showing up as lock waits on the same instance.
  • The DDL for the checkpoint table, which would show whether a step's result and its checkpoint commit in one transaction.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories