Green dashboard, dead switch: ndots:5 and a wildcard record aim your heartbeat at Traefik

The rule has a latency budget before anything goes wrong: 86400 seconds is 24 hours, and `for: 15m` adds a quarter hour, so the earliest this dead man's switch can page anyone is 24 hours 15 minutes after the last good push ^[1]. That is the intended cost of the pattern ^[4].

The published symptom and the published rule do not quite fit together. If Pushgateway is holding a timestamp that stopped advancing days ago ^[1], then `time()` minus that timestamp is far past 86400 and BackupTooOld should have fired ^[4]. Permanent silence only makes sense for a series that never existed at all: the new `ai-jobs` namespace that is not in Pushgateway's ingress allow-list ^[15] never lands a first push, so there is no sample for the expression to compare against ^[4]. A staleness alert covers jobs that used to work. It does not cover a job that has never once reported, which is exactly the state a fresh namespace starts in.

The DNS half is arithmetic. Four dots in `pushgateway.monitoring.svc.cluster.local`, a threshold of five, so the resolver treats the fully qualified name as relative and walks the search list first ^[7][8][2]. The search list in a default ClusterFirst pod has four entries, the last one inherited from the node ^[5][6], which puts the absolute name fifth in line, not fourth: without the wildcard you pay four wasted lookups per push, one more than the write-up counts ^[3][12]. With `*.lab.example.com` in place, the fourth query gets an answer, the answer is the ingress controller's LoadBalancer VIP, and the fifth query never happens ^[10]. The heartbeat is now addressed to Traefik on port 9091 ^[11].

Then the policy layer removes the evidence. The VIP is outside the pod and service CIDRs, so an egress policy scoped to in-cluster traffic drops the packet on the way out of the pod ^[14]. Most CNI configurations send no RST and no ICMP error for a NetworkPolicy deny, so curl hangs rather than fails ^[16], and the command as published carries `-s` and no timeout flag ^[3], which means nothing bounds the wait and nothing is written to the job log. The CronJob exits 0 ^[1].

Each layer would have been a five-minute fix in isolation; the stacking is what makes it durable ^[18]. And the choice of victim matters more than the mechanism. The same resolver behaviour breaking TLS validation produces a certificate error somebody sees; here the only casualty is the reporting path, which has no user-facing symptom by construction ^[17].