Build1 publisher3 min readPublished
Returning 202 with a persisted job row outlives nginx's sixty-second timeout
A dev.to post argues a report generation request should be accepted as work and written to a row with an owner, a budget and an idempotency key before any worker calls the model, so a browser retry collides with the row that already exists and reads its status.
The Engineer · Build desk
What happened
- A teammate clicked Generate on a report screen, nginx hit sixty seconds, and the gateway returned 504 while the model process carried on generating for nobody.
- The refresh that followed immediately spent a second free inference call on the same prompt, because the POST itself had been the product.
- The proposed replacement returns 202 Accepted with a job identifier, persists the prompt under the caller's identity, and lets a worker call the provider while the client polls or subscribes.
- The published schema puts status behind a four-value CHECK constraint and enforces UNIQUE (user_id, idempotency_key) on report_jobs, with daily spend tracked in a separate inference_budgets table.
- The FastAPI route is declared status_code=202 and requires an Idempotency-Key header alongside the bearer token before any provider call happens.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Adoption costs client work more than server work: every caller has to generate an Idempotency-Key and own a poll loop before the screen works again.
- capability A timeout after the insert costs the response but not the inference, because the retry reads the row's status instead of buying a second completion.
- exposure The debit lands before the row does, so a user whose insert or publish fails can lose daily tokens to a job that never runs.
- constraint Deduplication is scoped to user_id, so any deployment where many end users share one service account will collapse their distinct requests onto a single job row.
The handler the post proposes writes the work down before it asks for any. require_user rejects a junk bearer with 401; then the POST estimates tokens, looks for an existing job under the caller's id and Idempotency-Key, reserves budget, inserts a row with status queued, and publishes the job id to a broker topic named report.jobs [14]. The model call comes after all of that. The read side answers 404 when the job is missing or belongs to another user [15], so a poll loop cannot enumerate other people's job ids.
fetch_job comes before reserve_budget, so a double click or a React remount returns the existing job_id and status without debiting the budget a second time [3]. "If your table cannot say no, your provider will say yes twice," the post's author wrote [10]. The constraint behind that is UNIQUE (user_id, idempotency_key) [8]. Dedupe is per caller: two users can send the same key and get two rows, and many end users behind a single service-account user_id collide on one [6].
The budget half is thinner. estimate_tokens returns max(1, len(prompt) // 4) and the prompt field is capped at 8000 characters [12][13], so the biggest reservation a single call can make is 2000 tokens [1]. That figure is computed from the prompt before the worker runs, and nothing in the handler writes the completion's actual usage back [2]. Short prompts with long outputs will spend more at the provider than spent_tokens records [2]. The cap comes from DAILY_TOKEN_CAP in the environment [16]. Exhaustion returns 429 with error budget_exhausted and a retry hint of "tomorrow" [14]. The window is keyed on day_utc, so the reset follows UTC midnight wherever the caller sits [9][5].
reserve_budget, insert_job and broker.publish are three separate calls [14]. A failure between the first and the second debits a user's daily tokens for a job that never reaches the queue [4].
The post's answer to the free-tier objection is that a constrained or cold server is precisely when the request thread will 504, and a free path is precisely when the client retries without shame [18]. "Free tokens do not change the physics of a reverse proxy, and a free server does not make a blocked request thread honest," the author wrote [4].
The sixty-second wall in the anecdote is one deployment's proxy setting [1], and raising it just moves the wall further out. The 202 helps in another deployment only where the slowest hop between browser and worker has a read timeout below the slowest completion. Where completions land well inside that timeout, what the job row buys is dedupe and a spend record, and what it costs is an Idempotency-Key on every client plus a poll loop [5][11]. The author calls it "a proposed working slice, not a trophy benchmark" [7], and the evidence in the post is one 504 and two tables [17]. I would take the job row anyway on anything user-facing that calls a provider, because the server change ships more easily than the client change, and clients get harder to change once other people integrate against them.
What to watch
- Whether the worker side is published, including how a job stuck in 'running' is recovered when updated_at is the only timestamp it touches.
- Whether a follow-up reconciles token_estimate against provider-reported usage once the completion returns.
- Whether the slice grows a cancel path, since the status CHECK admits only queued, running, succeeded and failed.