Skip to content

Build1 publisher3 min readPublished

Bumping the lease epoch every five seconds gives a fenced write five seconds to commit

A dev.to worked example keeps HTTP clients and model SDKs out of the lease renewal path. Its Postgres fence rises on every renewal, so a write that outlives one five-second interval is rolled back by its own lease.

The Engineer · Build desk

Illustration accompanying Bumping the lease epoch every five seconds gives a fenced write five seconds to commit

What happened

  • A dev.to post argues that free inference belongs nowhere in lease renewal or leader election, naming latency, non-determinism and vendor authority as the three failure modes.
  • Its worked example is a Postgres table, worker_leases, holding name, holder, epoch and expires_at, with an index on expires_at, plus acquire and renew statements that return an integer epoch or nothing.
  • The author labels the schema and renewer a worked example rather than a multi-region consensus protocol, and tells readers to treat both as sample code.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure A team that routes membership through a vendor endpoint has to monitor refusal classes directly, because a dashboard of HTTP 200s will show the call succeeding while the vote never arrives.
  • constraint The pattern only ports to storage that can evaluate the epoch itself; a queue or object store with no view of the lease table cannot enforce the fence at the point of write.
  • cost The write path pays for the per-renewal bump: any mutation slower than five seconds arrives with an epoch the table has already replaced, and the application has to re-acquire and retry.
  • decision Keeping the ban on inference in the module's import list makes it a merge-time check on dependencies instead of a judgement call in code review.

Call renew() and exactly one statement runs. The UPDATE bumps epoch by one, pushes expires_at to now() plus the TTL, and matches only the row where the name and holder equal this process and expires_at is still in the future [9]. RETURNING epoch hands back an integer, or the cursor comes back empty. Empty means renew() returns None and hold() exits with "lost lease; refuse all writes" [11].

The defaults are TTL_SECONDS = 15 and RENEW_EVERY = 5, with the holder id built from hostname and pid and the DSN read from LEASE_DATABASE_URL [8]. Three scheduled attempts fall inside one TTL, at five, ten and fifteen seconds after the last success, so one failure leaves a second try and a third that arrives at the expiry boundary [1]. After a clean renewal there are ten seconds of margin the schedule does not intend to spend [2]. The post puts the renewal budget in hundreds of milliseconds and argues that a busy free endpoint can stall, queue or return a truncated object, so the lock expires while the response is still streaming [2].

The write path is tighter than the renewal path. append_order leaves the clock to SQL: the INSERT into orders selects only WHERE EXISTS a worker_leases row matching name, holder, the exact epoch passed in, and expires_at > now(), and a rowcount other than 1 triggers a rollback and RuntimeError("stale fence; drop the write") [12]. Because renew increments the epoch on every cycle, a mutation that starts under epoch N and reaches the EXISTS check after the bump is rejected by its own holder's lease: any epoch authorises writes only until the next renewal lands, roughly five seconds [3].

That guard works because the lease rows and the order rows sit in one database and one transaction. The post states the requirement plainly: "Downstream storage must reject any write whose epoch is stale" [13]. It adds that "A storage layer that cannot see the fence is not protected by the loop" [14]. Move orders into a queue or an object store and the EXISTS clause has no lease table sitting in the same transaction to test.

On authority, the argument is about who owns the refusal: "Leader election that depends on a vendor's refusal class has already given that vendor a veto over the write path," the post says [4]. The consequence it describes is a monitoring one, since the veto stays invisible in dashboards that only chart HTTP 200 [5]. On determinism, it says the same health snapshot can yield keep, drop, or a sentence that fails JSON, while a control plane needs the same inputs to produce the same fence [3]. The state being protected is holder id, epoch and expiry, and, as the post puts it, "Nothing in that tuple is improved by a temperature setting" [19].

The post asserts that teams keep stuffing chat completions into leader election, without naming a team, a product or an incident [18]. The author also labels the schema and renewer "a worked example, not a multi-region consensus protocol" [15].

The enforceable part is the file header, which tells the reader to keep lease_loop.py free of HTTP clients and model SDKs; the module imports os, socket, time and psycopg [7]. A closing artifact meant to fail CI when the lease package grows an inference client breaks off mid-word in the published text [17]. The post does leave inference one job: a completion that explains a failed insert is "optional theatre after the fact. It must not retry the insert, bump the epoch, or pick a new holder" [16].

What to watch

  • The CI artifact meant to fail when the lease package gains an inference client is cut off in the published text; whether it checks the import graph statically or at runtime decides how enforceable the rule is.
  • A published postmortem of a model-mediated leader election would move this from mechanism to incident. The post cites no such postmortem.
  • An append_order variant for storage outside Postgres would show whether the EXISTS guard survives when the lease table and the data live in different systems.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories