Build1 publisher3 min readPublished
Each background job retry gets its own Postgres row in a design built around tenant incident queries
Three separate IDs and one Postgres row per retry let an investigator list every attempt for a tenant in order, according to a dev.to design. Adopting it means carrying an operation ID into every enqueue call, and its sample redactor lets bearer tokens through.
The Engineer · Build desk

What happened
- A dev.to post on SaaS background jobs says every attempt should get its own ID and Postgres row, with retries appending evidence instead of overwriting it.
- The design keeps three identifiers apart: job_id for the logical work, attempt_id for one execution, and operation_id for the customer-visible action.
- The post rejects a single mutable jobs row because each update to last_error erases the error recorded for the attempt before it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Row count grows with every retry, and every enqueue path must carry the customer operation ID for the tenant query to find the job at all.
- exposure Teams that copy the sample redactor as written will keep bearer tokens and JSON-encoded secrets in the table on-call reads during incidents.
- decision Counsel and security owners have to set retention and erasure behavior for attempt rows, and the post warns against reading a universal period out of GDPR Article 17.
- capability A team can move from BullMQ or Agenda to a Postgres-backed cron worker without rewriting its incident procedure, provided failures are normalized at the worker boundary.
The schema starts from one incident question. Given a tenant ID, an operation ID and a time range, an investigator should find the job and read its attempts in order. They should also be able to tell a retryable failure from terminal exhaustion and reach sanitized diagnostic detail [4]. I think designing from the query is correct. It limits storage to what an investigation reads, and the post calls that goal narrower than retaining every log forever [4].
The separate identifiers exist for one failure mode. If a single ID stands in for job, attempt and operation, an operator looking at two errors cannot tell a duplicate execution from two retries, according to the post [5]. Its summary of that case is two words: "Ambiguity wins." [19]
Existing tools each hold a fragment. The worker library scheduled the retry and the error tracker grouped the exception, but the post says neither is an incident history by itself [3]. A column called last_error is at least honest about how much it remembers [8]. Logs fail for other reasons: retention differs between systems, clocks disagree, and a line can be emitted before the transaction that claims the work has committed [9]. So the post makes append-oriented attempt rows the durable record and keeps the job's current status as a compact projection for the dispatcher [10].
Write order is what makes the row trustworthy. The Python sample mints a UUID, then inside a transaction inserts the attempt with job_id, operation_id, tenant_id, attempt_no, started_at and an outcome of 'running' [15]. Each attempt costs two writes, the insert and the close, and a later attempt never touches an earlier row [16]. The post calls the attempt immutable [13]. In practice it is updated once, when it closes.
The open row matters when a worker crashes. If the insert commits before the handler starts, a worker that dies mid-attempt leaves a committed row at 'running' [17]. That row exists whatever the logs managed to flush [9]. Something outside the worker still has to find it and close it [17].
What goes into the row is bounded. It holds timestamps, attempt number, outcome, an error class, a bounded message and a correlation key into detailed telemetry, and no session tokens, connection strings or raw request bodies [6]. The post cites OWASP's logging guidance for both halves of the sanitizer: do not record tokens or passwords, and sanitize event data against log injection [7]. The sample's clean_error runs a case-insensitive regex for authorization, password or token followed by an equals sign, and replaces the value with [REDACTED] [14]. It then turns carriage returns and newlines into spaces and cuts the message to 500 characters [14].
The line-break handling is sound. The redaction is narrower than it looks. The pattern needs a literal equals sign after the key, so an exception message carrying `Authorization: Bearer <token>`, or a JSON body with a quoted "token" field, passes through intact [18]. Parameter binding keeps the SQL statement safe [13]. The regex is what decides whether the evidence table ends up holding secrets, and as written it covers query-string syntax only [18].
What to watch
- How a sweeper or lease timeout closes attempt rows left at 'running' by workers that crashed after the insert committed.
- Whether the redaction pattern is extended to header and JSON forms such as Authorization: Bearer and quoted token fields.
- Attempt-table growth per tenant during retry storms, and the retention period security owners settle on for those rows.