S3 to Lambda is async and at-least-once: the 3% that vanished after eight quiet months

The pipeline did not acquire a bug in month eight. It kept doing what it had always done, which was to treat Lambda's automatic retries as error handling, because until then the retries had quietly worked ^[14]. That holds while failures are transient. Larger files make failures deterministic: whatever the object size crossed, it gets crossed again on attempt two and again on attempt three, and then the event is discarded with no destination configured to receive it ^[1][2]. The account published on dev.to, originally on kuryzhev.cloud, does not name the limit that broke, and the arithmetic matters more than the limit anyway. One delivery plus two default retries is three chances, all spent on the same input that fails the same way ^[11]. The configurable range is 0 to 2, so the floor is a single attempt before the event is gone ^[1][12].

Note where the bytes come from. The event carries bucket, key, size, eTag and versionId, and the function calls GetObject itself ^[7]. File growth therefore lands inside the invocation, not inside S3's delivery, which is why a partner changing batch size is a change to your function's runtime behaviour and not to your integration.

The silence is the operationally expensive part. CloudWatch shows two failed invocations and then nothing, which reads exactly like a transient problem that resolved itself ^[4]. Roughly 3% of objects disappeared with no error, no alert and no trace ^[13], and the shape of that evidence is indistinguishable from success.

Two more assumptions compound it. S3 can pack multiple records into one invocation payload, and does so particularly under load ^[15], so a handler that reads event['Records'] as a single item starts losing data at exactly the moment volume rises ^[15]. Wrap that handler in one try/except that raises on the first bad record and five records become five losses when one is malformed, with the other four either reprocessed for nothing or gone if the retries also fail ^[9]. The blast radius of a single bad object is set by S3's batching decision, not by yours.

The fix in the post is structural rather than clever: iterate records independently and collect failures instead of raising on the first error, described as the single biggest change ^[5]. The sample pairs that with a conditional write to a DynamoDB table keyed on bucket, key and eTag, so a repeat delivery is a no-op ^[10]. That guard is not optional hygiene. At-least-once delivery means duplicate invocations for the same object are expected behaviour ^[6], and without dedup the visible symptom is duplicate rows, double-charged invoices or a revenue report that double-counts an afternoon ^[3].

Two other settings belong in the same design pass. Reserved concurrency, because a bulk sync of 10,000 files spikes concurrent invocations at once and an unbounded function either throttles the account or drops events it cannot absorb ^[16]. And the output prefix, because writing processed results back into the prefix that triggers the function loops, exhausts account concurrency in minutes, and produces a bill overnight ^[8].

The useful question about an S3 trigger is not whether the handler works. It is what artifact exists after it fails.