A retry that sends the email twice is a correctness bug, not a slow job

A newsletter from newsletter.systemdesign.one sets out why long-running agent runs fail badly: a workflow that searches dozens of documents, calls an LLM several times, extracts what it needs and assembles most of the final answer loses all of it when one step near the end fails ^[1]. The part worth taking seriously is not the wasted ten minutes. It is that if the agent already sent an email or updated a database before the failure, running the workflow again can repeat those actions ^[2].

That is the whole design problem, and the piece states it without hedging: a duplicate charge is not a retry problem, it is a correctness problem ^[9].

Four properties, according to the newsletter, separate an agent run from a self-contained background job. Steps are slow and expensive, since a single LLM call can take several seconds and cost real money, so replaying five finished steps to reach the sixth is not a rounding error ^[7]. Outputs are not deterministic, so rerunning a completed step does not guarantee the same result and a restart can change the workflow's outcome ^[8]. Some steps produce real external effects: an email sent, a record updated, a card charged ^[9]. And some steps wait on people, where an approval pause can run for hours or days, holding a worker process open for the duration does not scale, and a basic job queue is not designed to track where a multi-step workflow was waiting and resume it later ^[10]. Underneath all of it, one run chains LLM calls, tool calls, database reads and external API requests, each of which can fail on its own ^[6].

The worked example is a research agent that retrieves sources, extracts evidence, generates a draft, waits for editor approval, then publishes; if the model provider times out during drafting, retrieval and extraction are already done, and a system with no memory of progress restarts everything ^[11]. Put that next to non-determinism and the second attempt is not merely double the spend: the replayed extraction can return different evidence, so the recovered run is a different run ^[12].

Durable execution is the named remedy. It tracks progress as the workflow executes so a failure does not erase what came before, and a workflow whose first five steps succeeded resumes at step six ^[3]. The newsletter uses Inngest, which it describes as a durable execution platform for background functions, workflows and AI agents, as its case study ^[4]. It also draws the line between a chatbot, which answers and stops, and an agent, which keeps working toward a goal and decides what to do next as the work unfolds ^[5].

Be clear about what checkpointing buys. It buys resumption. It does not by itself make a send-email step safe to execute twice, because that depends on where the step boundary sits and whether the effect is recorded as committed before anything downstream can fail. The available text stops after the mid-workflow failure scenario and does not get to the mechanics ^[13], which is where the actual engineering cost lives.

Watch whether teams inventory which steps in an agent run touch the outside world, and whether those steps carry a caller-supplied key so a replay is a no-op rather than a second charge. Watch the delivery semantics vendors publish for step retries, since at-least-once execution pushes idempotency back onto the application. And watch how the human-approval pause is billed and scheduled, because a workflow parked for three days is a state-storage problem wearing a queue's clothing.