Loading today’s stories
Build1 distinct publisher3 min readUpdated
One operator flipped Cilium to full kube-proxy replacement and woke at 2:47 AM to a SIEM ingesting zero events. The cluster was healthy. The evidence pipeline was not.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
An operator upgraded Cilium on a four-node bare-metal cluster from partial kube-proxy replacement to full replacement, and the alert that eventually fired came not from the cluster but from his SIEM, whose ingestion rate for the `kubernetes-audit` and `network-flow` streams had flatlined to zero at 2:47 AM [1][2][7]. Every health signal stayed green through it, which is why this belongs in the change-review conversation and not the bug tracker [6][17].
The setup is a homelab: a Dell OptiPlex plus three Raspberry Pis running Talos Linux, Cilium, ArgoCD and Longhorn [1]. Before the change, Cilium 1.15 was in partial mode, with kube-proxy's iptables rules still doing the work for NodePort and ClusterIP [2]. The Cilium 1.16 notes called the eBPF replacement "production-ready for all environments," and the Talos documentation reduced the migration to one bullet point: set `kubeProxyReplacement: true` [3][4]. The actual change was a three-flag `helm upgrade` [5]. Pods rolled, nodes stayed up, `kubectl get pods -A` looked normal, and the author went to bed [6].
The mechanics explain the blindness. kube-proxy watches the API for Service and EndpointSlice changes and writes them into iptables as PREROUTING chains, a KUBE-SERVICES target and a DNAT to a backend pod IP [12]. Cilium instead attaches eBPF programs at `tc` hooks, looks the ClusterIP up in a BPF map keyed on address and port, rewrites the destination and forwards [14]. The author's own summary of the new path is "no iptables, no chains, no table reloads" [15]. That is the performance case and the observability liability in the same sentence: if your flow logging, host IDS or network agent reads conntrack tables or netfilter counters, the surface it reads has been removed. He calls the post an autopsy of deleting the datapath a security stack depends on, and by his headline the gap ran six hours [18][8].
What makes this worth attention is that observability was one of the stated reasons for the migration. iptables drops packets silently, with no log line and no metric, and identifying the offending rule means running `iptables -L -v -n` on every node [11]. Hubble was supposed to fix that with per-flow visibility [16]. It did, in the sense that Hubble showed flows after the cut-over [17]. But flows in Hubble are not events in the SIEM, and nothing in the one-bullet migration guide connects the two.
The performance case was real, at least at this scale: roughly 120 services generated about 3,500 iptables rules across the nodes, or around 29 rules per service, with every new service triggering a full `iptables-restore` that locked netfilter for hundreds of milliseconds [9][19][10]. On a Raspberry Pi that is a visible latency spike [10]. The old path also carried O(n) per-packet rule evaluation and no connection tracking visibility without conntrack tools [13].
Two things to check before you make this change. First, enumerate which of your log and metric streams actually derive from netfilter or conntrack, and confirm each has a Hubble-based or agent-based equivalent that is already exporting off-cluster. Second, treat SIEM ingestion rate as a cluster health signal, because in this account it was the only thing that noticed [7][17]. One caveat on the source: the published excerpt breaks off before naming the root cause or the fix, so the diagnosis is not available to copy [20].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The author ran a Cilium upgrade on a four-node bare-metal homelab cluster consisting of a Dell OptiPlex and three Raspberry Pis, running Talos Linux, Cilium, ArgoCD and Longhorn.
The Cilium 1.16 release notes described the eBPF kube-proxy replacement as "production-ready for all environments".
The Talos Linux docs covered the migration with a single bullet point: "Set kubeProxyReplacement: true in Cilium values."
After the upgrade the Cilium pods rolled, the nodes stayed up, kubectl get pods -A showed everything running, and the author went to bed.
At 2:47 AM the author was paged not by the cluster but by his SIEM: the log ingestion rate for the kubernetes-audit and network-flow streams had flatlined to zero events per second, a sharp drop rather than a gradual one.
In the author's homelab, roughly 120 services meant roughly 3,500 iptables rules across all nodes.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed but uncorroborated first-person account
The mechanism is described with unusual specificity — exact helm flags, the kube-proxy chain/DNAT path, the BPF service map layout, the hostNetwork/NodePort SNAT interaction — which raises plausibility. But this is a single self-published post with no command output, ingestion chart, restoration timeline or second publisher, the excerpt truncates mid-sentence before any conclusion or fix, and the referenced Cilium 1.16 release notes and Talos docs are quoted second-hand rather than supplied.
One homelab cutover against a vendor GA claim
The only concrete deployment evidence is a single four-node homelab cluster with roughly 120 services flipping to full eBPF kube-proxy replacement, plus a reported upstream GA declaration in Cilium 1.16. No production fleet, no other operators, and no usage or scale disclosure appear in the supplied material, so the described failure mode has one observed instance.
Production framing overstates a homelab anecdote
The title claims a replacement 'in Production' and a six-hour blackout, while the body describes a homelab and supplies neither a recovery timeline nor an ingestion chart; the performance summary ('O(1) lookup, sub-microsecond latency') is asserted without any benchmark. The underlying technical observation — that moving NodePort handling into eBPF can silently break hostNetwork log-ingestion paths while every health check stays green — is real and useful, so the gap is presentational inflation over a genuine finding rather than a fabricated story.
Practitioner post shaped by engagement, no vendor stake disclosed
The material discloses no vendor sponsorship, product or commercial relationship with Cilium, Isovalent or Talos; the author positions himself as an AWS practitioner writing up his own homelab. The observable incentive is audience attention on a developer publishing platform, visible in the 'in Production', '6 hours' and 'couldn't see a single packet' framing and the withheld root cause, which pulls presentation ahead of the evidence supplied. Absence of disclosed conflicts is not proof of none, so this is scored as mild rather than negligible distortion pressure.
Moderate on mechanism, low on magnitude
Confidence is moderate that the described interaction is real and instructive, because the mechanism is internally coherent and specific to identifiable components (NodePort handling moving to eBPF, SNAT with the node IP, hostNetwork collectors). Confidence is low on scope and magnitude: one publisher, one homelab, a truncated excerpt, no measurements, and an unverified six-hour figure.
build
Kubernetes Services scale in the dataplane, and every node keeps its own copy of the map1 distinct publisher
build
One alert, two causes, four green dashboards: the day the stack agreed and was wrong1 distinct publisher
build
Thirteen tasks green, then "give up (Recommended)" on the one that needed understanding1 distinct publisher
build
The network already knew: UPS and WAN state as keys a cluster can reconcile against1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 19, 2026