Retry Logic Is Guessing Until Attempts Are Logged And Operations Have Names

A field note from Loop & Retry, republished on dev.to as part of a series on LLM agents that survive production, argues that retry logic is guessing unless you can see what each attempt did [s1c1] [s1c2]. For anyone running a fleet with a retry budget, the operational consequence is that idempotency keys and per-attempt context stop being hygiene items you schedule later and become the things that make the budget mean anything [s1c2] [s1c14].

The failure mode is stated plainly: an attempt looks the same whether it is recovering from a transient network hiccup or whether the caller is stuck in a loop burning budget on a permanent failure [s1c2]. The post frames idempotency as a consequence of a deeper rule rather than a retry feature in itself: retries are only safe when the operation is idempotent, and idempotency is only verifiable when the operation carries a stable identity [s1c4]. An idempotency key is that identity, a token that tells the server to return the cached result rather than replay the work [s1c3]. Without it, a retry is not a retry, it is a second genuine execution [s1c5].

The construction rules are where this gets expensive. A UUID per task, shared by every attempt of that task, is the behaviour you want [s1c6]. A key that changes per millisecond means two retries 100ms apart present different keys and the server double-processes [s1c7]. The stability window has to cover the retry window: retry for up to five seconds and the key must stay stable for five or more [s1c8]. The post's own sample code then builds the key from user id, resource id, operation name and `int(time.time() * 1000)`, with a comment claiming stability within roughly one second [s1c9]. A millisecond term does not do that; any two attempts more than a millisecond apart get distinct keys, which is exactly the double-processing the prose warns against [s1d1]. Read the code, not the comment, in your own repos too.

Identity solves deduplication. Correlation solves debugging. The second pattern carries a `logical_op_id` that is stable across all attempts of one operation, and logs each attempt with the attempt number, latency in milliseconds, status, error and timestamp [s1c10] [s1c11]. That is two separate stable identifiers per operation plus an attempt counter, and they are not interchangeable [s1d2]. The payoff is discriminating between failure shapes: three attempts returning the same "rate limit: retry after 60s" is a real limit, not a transient, while a timeout followed by a success is evidence the retry actually earned its cost [s1c12]. Instead of "failed: 500", the trace reads attempt 1 timeout after 3.2s, attempt 2 timeout after 2.8s, attempt 3 upstream rate limit [s1c13], which is six seconds of wall clock spent before the informative error arrived [s1d3].

Two things to watch. First, check whether your key stability window is derived from your retry window or from whatever clock call was nearest [s1c8] [s1c9]. Second, the cost accounting section is the part that matters for fleets, distinguishing total attempts from retries that led to success, retries that failed, and abandoned work [s1c15]; the version we were supplied cuts off mid-class, so the abandonment metric is unspecified in the material [s1c16].