A governance tag push turned five EKS workers into Prometheus scrape targets

The add-only rule was supposed to prevent this class of surprise. `techteam` was left unset at the ASG level because the rule said to use existing values only, and the EKS nodegroup ASG did not have one ^[4]. The running workers came up with `techteam`, `Application`, `environment` and `businessunit` anyway. The write-up attributes that to EKS nodegroup or launch template tagging tied to the same governance work, not to anything overwriting an older value ^[5]. The guard sat on the ASG script, and the tags reached the instances by a different path.

EC2 service discovery re-queries the tags on every refresh. It asks EC2 which instances carry the tags, then scrapes port 9273 on whatever comes back ^[6]. Writing a tag is a change to the scrape topology. EKS workers are EC2 instances, and they matched the filter as soon as the tags landed ^[7]. Those nodes report through in-cluster DaemonSets or cAdvisor, and nothing on them listens on 9273 ^[8], so the five new targets could only ever report `up=0` ^[9].

"From a compliance perspective, that looked fine. From a monitoring perspective, we had just told Prometheus these were prod platform hosts," the author wrote ^[14].

Five of the roughly nine down telegraf targets came from the tag write, about 56 percent of the batch ^[18]. Two were a real break, on new AL2023 instances where the telegraf RPM was installed but the `prometheus_client` output was commented out, so the agent logged "no outputs found" and had nothing to expose on the scrape port ^[11]. Fixes for AL2023 telegraf were already in the ansible repo; those hosts had no ansible-pull cron, so ansible-pull never ran to template `telegraf.conf` ^[12]. The account itemises seven of the nine ^[19].

The kafka target is a third defect firing under the same alert name. Its exporter on 9308 had been disabled for months while telegraf on 9278 on the same host was fine, and by the author's account the alert rule conflated exporter health with agent health ^[13]. "I started grouping targets by hostname and role before touching anything, which turned out to be the only reason we didn't make things worse," the author wrote ^[15]. Nine criticals arrived under one rule name on the Thursday morning, three days after the tagging ^[17].

For this to reach another fleet, the discovery job has to key on the same tag names the finance and compliance schema writes, and the Kubernetes nodes have to sit in the accounts and regions the tagging run covers. A job keyed on a tag the monitoring team owns would have kept the same targets through the run. The cheap exclusion was already on the instances: the nodegroup ASG carried `eks:cluster-name`, `eks:nodegroup-name` and `k8s.io/cluster/*` before September 8 ^[3]. A drop rule on `eks:cluster-name` in the EC2 SD config keeps those hosts out of the pool with no change to the governance schema. The five worker targets were still unfixed when the account was captured ^[16].