BuildNot yet confirmed elsewhere1 publisher2 min readPublished
Green dashboard, dead switch: ndots:5 and a wildcard record aim your heartbeat at Traefik
Two sane Kubernetes defaults plus one convenient DNS record send a backup job's heartbeat to an ingress VIP, where a NetworkPolicy eats it without a packet of complaint.
The Engineer · Build desk
What happened
- A backup CronJob exits 0, the Pushgateway heartbeat metric has not advanced in days, and Prometheus fires nothing at all.
- Default ClusterFirst pods get a resolv.conf whose search list ends in the node's own internal domain, with options ndots:5.
- Most CNI setups send no RST and no ICMP error for a policy deny, so the client hangs instead of failing.
Why it matters
- exposure The casualty is the reporting path itself, which by construction has no user-facing symptom, so every outage the heartbeat was bought to catch is now uncovered too.
- contradiction A stale-but-present series would have paged after 24h15m, so unbroken silence points at a series that was never created; that changes the thing to test from the last push to the first.
- constraint With an allow-list that enumerates namespaces, standing up a new one becomes an edit in the monitoring namespace, and a team that owns only its own namespace cannot make its own heartbeat arrive.
- decision The wildcard record is the piece that turns a slow lookup into a confidently wrong one, so per-app hostname convenience now has to be weighed against in-cluster names resolving where they claim to.
The rule has a latency budget before anything goes wrong: 86400 seconds is 24 hours, and `for: 15m` adds a quarter hour, so the earliest this dead man's switch can page anyone is 24 hours 15 minutes after the last good push [19]. That is the intended cost of the pattern [4].
The published symptom and the published rule do not quite fit together. If Pushgateway is holding a timestamp that stopped advancing days ago [1], then `time()` minus that timestamp is far past 86400 and BackupTooOld should have fired [4]. Permanent silence only makes sense for a series that never existed at all: the new `ai-jobs` namespace that is not in Pushgateway's ingress allow-list [15] never lands a first push, so there is no sample for the expression to compare against [21]. A staleness alert covers jobs that used to work. It does not cover a job that has never once reported, which is exactly the state a fresh namespace starts in.
The DNS half is arithmetic. Four dots in `pushgateway.monitoring.svc.cluster.local`, a threshold of five, so the resolver treats the fully qualified name as relative and walks the search list first [7][8][20]. The search list in a default ClusterFirst pod has four entries, the last one inherited from the node [5][6], which puts the absolute name fifth in line, not fourth: without the wildcard you pay four wasted lookups per push, one more than the write-up counts [22][12]. With `*.lab.example.com` in place, the fourth query gets an answer, the answer is the ingress controller's LoadBalancer VIP, and the fifth query never happens [10]. The heartbeat is now addressed to Traefik on port 9091 [11].
Then the policy layer removes the evidence. The VIP is outside the pod and service CIDRs, so an egress policy scoped to in-cluster traffic drops the packet on the way out of the pod [14]. Most CNI configurations send no RST and no ICMP error for a NetworkPolicy deny, so curl hangs rather than fails [16], and the command as published carries `-s` and no timeout flag [3], which means nothing bounds the wait and nothing is written to the job log. The CronJob exits 0 [1].
Each layer would have been a five-minute fix in isolation; the stacking is what makes it durable [18]. And the choice of victim matters more than the mechanism. The same resolver behaviour breaking TLS validation produces a certificate error somebody sees; here the only casualty is the reporting path, which has no user-facing symptom by construction [17].
What to watch
- Whether the full write-up names a third failure layer beyond DNS and NetworkPolicy; the supplied text is cut off mid-sentence at the hanging curl.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap+12
- Incentives28
- Confidence62
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
A CronJob exits 0, Pushgateway shows a heartbeat metric that has not updated in days, Prometheus fires nothing, and every dashboard is green while the guarantee the heartbeat existed to provide has stopped being true.
- [2]
The batch heartbeat pattern: Prometheus cannot scrape a pod that lived for forty seconds, so the job pushes a timestamp to Pushgateway when it finishes, Prometheus scrapes Pushgateway on its normal cycle, and an alert fires if the timestamp goes stale.
- [3]
The push is the last line of the backup script: echo "backup_last_success_timestamp $(date +%s)" piped to curl -s --data-binary @- "http://pushgateway.monitoring.svc.cluster.local:9091/metrics/job/nightly-backup".
- [4]
The alert rule is BackupTooOld, expr: time() - backup_last_success_timestamp > 86400, for: 15m, severity warning.
- [5]
A pod with the default dnsPolicy ClusterFirst gets resolv.conf with search ai-jobs.svc.cluster.local svc.cluster.local cluster.local lab.example.com, nameserver 10.96.0.10, options ndots:5.
- [6]
The kubelet appends the node's own search domains after the cluster ones, so if nodes carry an internal domain such as lab.example.com, every pod inherits it.
- [7]
The ndots:5 option means any name with fewer than five dots is treated as relative, and the resolver walks the search list before trying the name as-is.
- [9]
The generated query order is the name appended to ai-jobs.svc.cluster.local, then svc.cluster.local, then cluster.local (all NXDOMAIN), then appended to lab.example.com, which returns an answer.
- [10]
A wildcard record *.lab.example.com pointing at the ingress controller's LoadBalancer VIP, a common homelab setup for TLS and per-app hostnames, matches the fourth query and returns the ingress VIP; the resolver stops, so the absolute name that resolves to the real Pushgateway ClusterIP is never queried.
- [12]
Without the wildcard, the fourth query would return NXDOMAIN, the resolver would fall through to the absolute name and everything would work, slower and with three wasted queries; the wildcard converts a harmless inefficiency into a wrong answer.
- [13]
The monitoring namespace runs default-deny ingress, and Pushgateway has an allow rule for Prometheus scrapes and for the namespaces that existed when the policy was written.
- [14]
The LoadBalancer VIP sits outside the pod and service CIDRs, so if the workload namespace has an egress policy scoped to in-cluster traffic, the packet is dropped on the way out.
- [15]
Even after DNS is fixed, a newly added namespace such as ai-jobs for LLM batch work is not in Pushgateway's ingress allow-list, so its pushes are dropped on arrival.
- [16]
NetworkPolicy denies do not send RST packets or ICMP errors with most CNI configurations, so the drop is silent by design and the curl hangs.
- [17]
The author has written about the same resolver mechanism corrupting TLS validation, and says the Pushgateway variant is nastier because the failure has no user-facing symptom at all.
ReportedSupportedSource: dev.to write-up by the author of the earlier wildcard DNS and TLS postView cited source - [18]
Three independent failures stack and each one masks the next; any single layer would have been a five-minute fix.
- [19]
The rule cannot page before 24 hours 15 minutes after the last successful push.
- [20]
With ndots:5, the four-dot service FQDN is below the threshold and is treated as relative; it would need a fifth dot to be tried as written first.
- [21]
The staleness expression only crosses its threshold where the series has samples, so a path broken before the first successful push yields no series and no alert, which is the reading consistent with permanent silence.
- [22]
The search list has four entries, so the absolute name is the fifth query and the no-wildcard path costs four wasted lookups per push, one more than the write-up's count of three.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toPushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts
1 article · August 21, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.