Two thousand microVMs per host pushed Lambda's flow log past 100,000 iptables rules

iptables walks its rules more or less linearly for every packet, and each new microVM piled more rules onto the chain ^[7]. So a packet's bookkeeping cost rose with how crowded the host was and which slot the microVM received ^[9]. Divide well over a hundred thousand rules by a couple of thousand sandboxes and you get roughly 50 rules per microVM ^[8][17]. A function that sent two packets still paid to walk past its neighbours.

The other half of the inherited design was a userspace daemon that read the kernel counters, batched them into records, serialized them, and uploaded the files frequently ^[5]. That loop has to close before the evidence disappears. Thousands of microVMs on one server run for a few hundred milliseconds and then shut down, each one a different customer's function ^[2]. "Within milliseconds, the workload is done, and the logs captured during those few milliseconds are the only witness left," the AWS Lambda authors wrote in The New Stack ^[12][20].

Density was at least a tuning problem, but the borrowed kernel module did not speak IPv6 ^[10]. "A record that can't see half the address space isn't one you can trust, and the moment dual-stack IPv6 support was proposed for Lambda, the old approach was finished," they wrote ^[11].

That hundred-thousand-rule figure means something on someone else's fleet only if per-flow attribution keys on match rules in a table that is walked for every packet ^[7], and only if density runs into the thousands of tenants per host ^[2]. At about 50 rules per workload, a host running 30 containers carries roughly 1,500 rules ^[21], and that walk will not show up in a CPU budget. What transfers to a smaller fleet is the failure signature, a capture path whose cost tracks host occupancy.

The published account stops before the new pipeline's internals. It names the replacement as a purpose-built pipeline written in eBPF and Rust ^[13], and lists the non-negotiable design considerations as correct attribution, meaningful overhead reduction, and a third item the text cuts off after the words "support for I" ^[16]. The requirements are stated plainly enough to copy: the record must be complete and correctly attributed, and capturing it must add almost no overhead to the network flow or the platform ^[3]. Correct attribution means every flow ties to the microVM where it landed and the tenant that produced it, and completeness means no missed packets ^[18]. Those same records feed network usage and metering services, and they are persisted for audit and compliance ^[4]. AWS says a missing or misattributed record can cause billing, observability and monitoring problems at scale, on a platform serving millions of requests per second ^[15][19].