Skip to content

Build1 publisher3 min readPublished

Each background job retry gets its own Postgres row in a design built around tenant incident queries

Three separate IDs and one Postgres row per retry let an investigator list every attempt for a tenant in order, according to a dev.to design. Adopting it means carrying an operation ID into every enqueue call, and its sample redactor lets bearer tokens through.

The Engineer · Build desk

Illustration accompanying Each background job retry gets its own Postgres row in a design built around tenant incident queries

What happened

  • A dev.to post on SaaS background jobs says every attempt should get its own ID and Postgres row, with retries appending evidence instead of overwriting it.
  • The design keeps three identifiers apart: job_id for the logical work, attempt_id for one execution, and operation_id for the customer-visible action.
  • The post rejects a single mutable jobs row because each update to last_error erases the error recorded for the attempt before it.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Row count grows with every retry, and every enqueue path must carry the customer operation ID for the tenant query to find the job at all.
  • exposure Teams that copy the sample redactor as written will keep bearer tokens and JSON-encoded secrets in the table on-call reads during incidents.
  • decision Counsel and security owners have to set retention and erasure behavior for attempt rows, and the post warns against reading a universal period out of GDPR Article 17.
  • capability A team can move from BullMQ or Agenda to a Postgres-backed cron worker without rewriting its incident procedure, provided failures are normalized at the worker boundary.

The schema starts from one incident question. Given a tenant ID, an operation ID and a time range, an investigator should find the job and read its attempts in order. They should also be able to tell a retryable failure from terminal exhaustion and reach sanitized diagnostic detail [4]. I think designing from the query is correct. It limits storage to what an investigation reads, and the post calls that goal narrower than retaining every log forever [4].

The separate identifiers exist for one failure mode. If a single ID stands in for job, attempt and operation, an operator looking at two errors cannot tell a duplicate execution from two retries, according to the post [5]. Its summary of that case is two words: "Ambiguity wins." [19]

Existing tools each hold a fragment. The worker library scheduled the retry and the error tracker grouped the exception, but the post says neither is an incident history by itself [3]. A column called last_error is at least honest about how much it remembers [8]. Logs fail for other reasons: retention differs between systems, clocks disagree, and a line can be emitted before the transaction that claims the work has committed [9]. So the post makes append-oriented attempt rows the durable record and keeps the job's current status as a compact projection for the dispatcher [10].

Write order is what makes the row trustworthy. The Python sample mints a UUID, then inside a transaction inserts the attempt with job_id, operation_id, tenant_id, attempt_no, started_at and an outcome of 'running' [15]. Each attempt costs two writes, the insert and the close, and a later attempt never touches an earlier row [16]. The post calls the attempt immutable [13]. In practice it is updated once, when it closes.

The open row matters when a worker crashes. If the insert commits before the handler starts, a worker that dies mid-attempt leaves a committed row at 'running' [17]. That row exists whatever the logs managed to flush [9]. Something outside the worker still has to find it and close it [17].

What goes into the row is bounded. It holds timestamps, attempt number, outcome, an error class, a bounded message and a correlation key into detailed telemetry, and no session tokens, connection strings or raw request bodies [6]. The post cites OWASP's logging guidance for both halves of the sanitizer: do not record tokens or passwords, and sanitize event data against log injection [7]. The sample's clean_error runs a case-insensitive regex for authorization, password or token followed by an equals sign, and replaces the value with [REDACTED] [14]. It then turns carriage returns and newlines into spaces and cuts the message to 500 characters [14].

The line-break handling is sound. The redaction is narrower than it looks. The pattern needs a literal equals sign after the key, so an exception message carrying `Authorization: Bearer <token>`, or a JSON body with a quoted "token" field, passes through intact [18]. Parameter binding keeps the SQL statement safe [13]. The regex is what decides whether the evidence table ends up holding secrets, and as written it covers query-string syntax only [18].

What to watch

  • How a sweeper or lease timeout closes attempt rows left at 'running' by workers that crashed after the insert committed.
  • Whether the redaction pattern is extended to header and JSON forms such as Authorization: Bearer and quoted token fields.
  • Attempt-table growth per tenant during retry storms, and the retention period security owners settle on for those rows.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories