Cilium's kube-proxy replacement empties the chain your Service runbook greps

A ClusterIP is not an address on any machine ^[10]. When a Pod sends a packet to one, something in the forwarding path rewrites the destination to a real Pod IP, and kube-proxy was the component that registered those rewrite rules in netfilter, the Linux kernel's packet-processing framework ^[10]. iptables is the command that reads and writes netfilter rules ^[10]. Because the forwarding decision lived in the kernel as a text rule, dumping everything was enough to find it ^[11]. On the kube-proxy cluster the chain sequence runs KUBE-SERVICES to KUBE-SVC-xxxx to KUBE-SEP-xxxx, and the destination 10.96.111.7:80 becomes 10.244.0.6:80 at the last step ^[12].

The same structure is what made tcpdump useful. Early in that path the packet still carries the ClusterIP as its destination, so a capture filtered on the ClusterIP caught it, and capturing before and after the rewrite showed which Pod IP the destination changed to ^[13]. Rules from iptables-save, live packets from tcpdump: those two tools covered Service forwarding ^[14]. Under Cilium's kube-proxy replacement both go quiet. KUBE-SERVICES, the chain kube-proxy creates as the entry point for ClusterIP traffic, does not exist ^[7], and the author reports that tcpdump on a ClusterIP produces nothing and conntrack -L returns no matches ^[15].

A loose grep is worse than an empty one. Widen the pattern to KUBE- and lines come back, but they are KUBE-FIREWALL and KUBE-KUBELET-CANARY, kubelet's chains, and the nat table itself still holds rules ^[8]. All 13 KUBE-SERVICES lines are gone, and none of the KUBE- chains still present has anything to do with Service forwarding ^[18]. An on-call engineer who stops at seeing rules in iptables has read a populated table and learned nothing about where the packet went.

Two conditions decide whether that result transfers. The first is versions: kind v0.27.0, node image kindest/node:v1.32.2, Cilium 1.17.3, cilium-cli v0.20.0, kernel 6.8, both clusters single-node, with multi-node-only behaviors out of scope ^[16]. The second is that kube-proxy is genuinely absent. The test cluster used kind's kubeProxyMode: none, so it was never installed ^[5], while on a running cluster kubeProxyReplacement=true only tells Cilium to assume absence, and you have to remove kube-proxy yourself ^[4]. I would expect the half-migrated cluster to be the harder one to debug: the dump still returns chains, and Cilium is the thing forwarding the Service ^[3].

So the runbook edit belongs before the migration, while every Service is healthy and nobody is timing you. The article says it documents where Cilium keeps the forwarding rules that used to live in iptables, and what to check in what order when connectivity fails ^[15]. The published reproduction runs two kind clusters side by side, one with kube-proxy and one with Cilium, and issues the same commands against the same Service on both ^[17].