Build1 publisher3 min readPublished
Lambda durable function replay turned a human-approved refund into an escalation
One developer's Lambda durable function executed 'escalate' after a human approved 'refund', because replay ran the uncheckpointed agent call a second time. Whatever a human signs off on has to be checkpointed with context.step so every replay returns the same value.
The Engineer · Build desk
What happened
- When a durable function resumes, Lambda re-runs the handler from the top and injects the checkpointed result for each context.step in place of running it again.
- The author's instrumented suspend-and-resume counted two executions of a line outside any step and one execution of a line inside a step.
- Durable functions, released in late 2025, let a single handler pause for up to a year on a human or external event without paying for idle compute.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure The handler returns the proposal under "approved_proposal", so a replayed, uncheckpointed proposal would enter the execution record labelled as the action the human approved.
- cost Each replay repeats an uncheckpointed model call, so a handler with two approval waits would ask the model three times per workflow and can get three different answers.
- decision Porting an approval gate off Step Functions moves checkpoint boundaries out of a state machine definition and into handler code, where reviewers now have to catch a missing context.step call.
Replay exists because of where the resume lands. After a wait that might last a week, the handler comes back on a fresh container [6]. The local variables from the first pass no longer exist, and checkpoints are the only state that survives. The author of the dev.to post put the contract in two sentences: "Anything inside a step runs once and is remembered. Anything outside a step runs again on every replay." [7]
Put the approval gate through that contract. On the first pass the agent proposes, notify_human sends the callback ID to an approver, and wait_for_callback suspends the execution [5][11]. When the answer arrives, the handler starts again from its first line. If the proposal call sits outside a step, the agent is asked a second time. "Ask it twice, you can get two different answers," the author wrote [9]. The author calls that layout "the version most people write first" [10].
The value at risk in that sequence is the proposal. In the author's failure the approval came back intact: the gate opened and the system acted [1]. Its input was the part that moved. The author describes the remedy as a "one-line structural fix" [12], and the working handler in the post shows the layout, with the agent call inside context.step under the name "agent_proposes" [11]. Other code in that handler stays outside any step and is safe. The json.loads on the callback payload parses the same bytes to the same value on every pass [11]. The test for each line outside a step is whether a second run can return something different. A model call can, and so can a clock read or a generated ID.
The instrumented run is a single suspend-and-resume with one counter on each side of the step boundary [8]. It tests a contract, and a contract holds for any handler built on the same two primitives; the author says the behaviour "is stated plainly the moment you go looking" [15]. The workflow ran locally with no AWS account and was also deployed to real Lambda [13]. The post does not say which environment produced the counts.
The design deserves the credit the author gives it. Before durable functions, this gate meant a Step Functions state machine, task tokens threaded between states, and JSONPath to move data [4]. Durable functions reduce it to one handler and two primitives [5], and the author called the result "genuinely lovely" [14]. I think re-execution with checkpoints is the right way to rebuild state on a container that did not exist when the wait began. Its failure mode is quiet. Of the broken version, the author wrote: "The code looked correct." [16]
What to watch
- Whether notify_human, the function that sends the callback ID to the approver, runs again on replay and sends the approver a second request.
- Counts from the deployed Lambda run, which would confirm the local runtime replays the same way the service does.
- Whether AWS adds a check to the durable functions SDK that flags non-deterministic calls made outside context.step before a wait.