Build1 distinct publisher3 min readPublished
Adyen traced sandbox setup failures on its big data clusters to a garbage collector that walks every tracked connection in order, which turns the table-size knob you raised for headroom into part of the fault.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
The rate is the part that transfers. Adyen reports a maximum of 200,000 entries per second through the connection tracking table [4], which prices one entry at roughly five microseconds [16]. Seven million entries at that rate is 35 seconds, and sixteen million is up to 80 [4][5]. Entry count grew 2.29x between those two measurements and time grew 2.29x [17], so across the range they measured there is no amortisation, no early exit, no index. It is a walk.
That walk sits in front of the API the CNI plugin calls. On affected nodes, the durations of PUT /v1/endpoint and DELETE /v1/endpoint climbed linearly from 4:20pm and never recovered, meaning endpoint creation and deletion never finished [6]. The plugin gave up with a client timeout [2], and once one pod failed that way, every subsequent pod on that node failed too [3]. The connections being enumerated belong to pods that exited hours earlier [15].
What the published excerpt does not show is the serialization point: which lock, or which single-threaded loop, puts a garbage collection pass ahead of an endpoint create. Adyen says the agent logs had nothing even at debug level [7], and that the finding came from tracing syscalls, reading codebases and analysing eBPF internals [8], against Kubernetes 1.31.7 and Cilium 1.16.5 [9]. Read the causal chain as the authors' reading of their own traces.
The tuning list is the uncomfortable part. Before hitting this, Adyen had already raised DNS proxy timeouts, expanded the connection tracking table capacity, raised API rate limits and streamlined security labels [13]. Expanding the table is the standard answer to eviction under connection churn. On a linear scan it is also an increase in the worst-case stall, because maximum scan time is entries divided by 200,000 [19]. The knob that stops connections being dropped raises the ceiling on how long a node can refuse to start pods.
For the numbers to transfer you need Adyen's shape. Their Trino pods pull files from more than 500 HDFS datanodes, one connection per file, about 50,000 connections a minute per pod at peak [11], which is roughly 833 a second [18]. Batch pods live anywhere from a second to a couple of hours [12]. Nodes are 64 physical cores with 512GB of RAM, or 128 cores with 2TB [10], so many of those pods share one kernel's table. If your pods hold pooled connections to a dozen services and your nodes have eight cores, your table stays small and the walk is milliseconds. This failure needs a large node plus a workload that treats a TCP connection as disposable.
In my context I would cap the table and accept eviction rather than raise it and accept a minute-plus window where a node cannot admit pods [5]. That preference depends entirely on having retry-tolerant clients. If a reset means lost data on your path, the trade runs the other way, and you are back to sizing nodes so that no single kernel table gets large enough to matter.
Ranked by verification strength, evidence, and original report placement.
Adyen runs Cilium CNI across all of its 100+ Kubernetes clusters, having switched from Calico.
Pods failed with: 'Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox ... plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded'.
After the first timeout on a node, any other pod attempting to spawn on that same node also timed out.
Adyen reports 35 seconds as the latency required to iterate through a connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second.
At Adyen's 16-million-entry peak, the sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts that prevented new pods from starting on affected nodes.
On affected nodes the API duration for DELETE /v1/endpoint and PUT /v1/endpoint increased, with endpoint call latency rising linearly from 4:20pm onwards, indicating that endpoint creation and deletion never finished.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · August 30, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Deleting kube-proxy moves your service traffic where your SIEM cannot follow it1 distinct publisher
build
Cross-node pod traffic: routing or encapsulation, and why that choice is a debugging decision1 distinct publisher
build
Kubernetes Services scale in the dataplane, and every node keeps its own copy of the map1 distinct publisher
build
Kubernetes 1.35 lands 60 enhancements, 17 of them stable: treat the upgrade as a compatibility audit1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Checkable in the details, silent on the cure
The specifics are unusually falsifiable for a company blog: a quoted CRI error, two named software versions offered precisely so readers can read the same code, endpoint API routes identified by path, and latency figures whose ratio holds to within a rounding error. What keeps this short of strong is that the account ends before the remedy, cites no upstream issue or patched release, and offers the 80-second peak as arithmetic from a high-water mark rather than a timed stall.
One large fleet, no trail beyond it
The deployment behind the finding is real and sizeable — Cilium on every one of 100-plus clusters, nodes up to 128 cores and 2TB, one Trino pod opening some 833 connections a second. But adoption of the finding itself is where the score sits, and there it is a single operator: nobody else in this reporting confirms the stall, and no Cilium release, issue or advisory is shown responding to it.
Headline arithmetic, cautious footing
The lede leans on 80 seconds, which is the 16-million-entry peak divided by the 200,000-per-second ceiling rather than an observed stall, and the framing that hours-dead pods block live ones is written for effect. Set against that, the impact paragraph is disciplined — analytics only, resolved before any SLO breach, payments untouched — and the piece never claims a general Cilium defect. Mild overstatement, mostly in the presentation of a computed worst case.
Recruiting-grade engineering writing, no product to move
Adyen sells payments, not networking, so it has no stake in how Cilium is perceived and nothing to upsell here; the visible motive is engineering reputation. The one place interest bites is the impact paragraph, where the party with the most to lose from a payments-outage reading is also the sole source for the claim that no transaction flow was touched.
Solid on mechanism, open on scope
We can be fairly confident about what happened on those nodes and why the scan cost scales the way it does — the numbers are internally consistent and the versions are pinned. We are much less confident about how far it generalises, since a single company's postmortem, published without upstream confirmation or the resolving configuration, cannot tell us whether this is a Cilium property, a tuning consequence, or a peculiarity of 2TB nodes running Trino against 500 datanodes.