Build1 publisher3 min readPublished
Leaving group_by unset makes every label combination its own Alertmanager notification
A dev.to walkthrough traces 3 a.m. pager storms to the route block, where grouping sits at the tool's default and no inhibition rule exists to suppress the symptoms of a node that has already gone down.
The Engineer · Build desk
What happened
- A dev.to post opens on a pager firing dozens of notifications at 3 a.m. for one disk-pressure event on a single node, routed to three teams that all subscribe to the node exporter alert.
- Keeping alertname in group_by stops alerts with different names from ever merging, so DiskPressure and MemoryPressure on the same node page separately however many labels they share.
- The post states plainly that Alertmanager cannot synthesise a root-cause summary, and that the best available outcomes are one batched notification of related alerts or fewer notifications through inhibition.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Whoever owns the group_by line decides what a notification means: drop alertname and the on-call gets one group per node containing mixed alert types, keep it and the same node fires once per alert name.
- constraint Routing work will not produce the root-cause message an engineer wants to read, so any correlation beyond batching and suppression has to be built outside Alertmanager.
- cost Three of the four fixes are lines in one file and can ship in an afternoon; the severity taxonomy costs a pass over every alert rule in the repository, which is why it tends to be the one left half-done.
- exposure A half-backfilled severity label set leaves a routing tree that pages correctly for some alerts and silently misroutes the rest, and nobody notices because the config looks finished.
The `group_by` line is where the trade-off actually gets made. While `alertname` is in `group_by`, alerts with different names never merge into one notification no matter how many other labels they share, so DiskPressure and MemoryPressure on the same node arrive separately [6]. The post's fix is to drop `alertname` and group on scope labels such as `cluster` and `instance`, and it is explicit about the price: alerts of different types on the same node then land in one group [7].
Inhibition is the second edit. Without `inhibit_rules`, a NodeDown alert and its downstream symptoms - pod restarts, service unavailable, high latency - fire in parallel, instead of NodeDown suppressing them [8]. The post is careful about what this buys you. Alertmanager cannot synthesise a root-cause summary; the realistic outcomes are one batched notification containing many related alerts, or fewer notifications overall through inhibition [9].
Three of the four root causes the post lists are lines in `alertmanager.yml`: the grouping keys, the missing inhibition block, and the timing intervals [18]. The fourth is a rule-file change. Severity labels are set in the alert rules, so a team that never agreed a taxonomy and stamped `severity: warning` on everything has a routing tree that cannot separate "wake someone up" from "check this during business hours" [10]. Fixing that means a pass over the rule files. Kuryzhev warns about the teams that start and stop halfway, leaving half the alerts routing correctly, "which is worse than having no severity labels at all because it looks fixed when it isn't" [11].
The timing cause is the quiet one. If `repeat_interval` is shorter than the real mean-time-to-resolve for an alert class, the same unresolved incident re-notifies every hour and gets filed mentally as spam [12]. A child route can override `repeat_interval` independently, but it still inherits `group_interval` from its parent unless that is set explicitly too [13].
The evidence is a walkthrough. The failing fleet is a hypothetical - fifteen exporters across sixty nodes, a `route` block copied from a tutorial two years ago with a single receiver and no inhibition rules [14] - and the post gives its fix as a partial config [16]. The post does not give notification counts. So the claim that alert fatigue is "commonly a routing configuration problem, not a Prometheus rule problem" [3] rests on how the defaults behave rather than on a measured incident. That part checks out on the documented behaviour: with `group_by` unset, Alertmanager groups by all labels, which treats every distinct label combination as its own notification [5].
What would have to be true for this to transfer to your cluster is that your correlated alerts share a scope label you are willing to group on. If DiskPressure and MemoryPressure carry `instance` and your on-call can triage from a per-node group, the edit is small. If your alerts arrive with different label sets from different exporters, dropping `alertname` merges less than the post's example suggests, and the noise that remains is in the rules [15].
What to watch
- Whether the post's full config sets group_interval explicitly on child routes, or leaves the inheritance trap it describes in its own example.
- Any published before-and-after notification counts from a real incident on the same four fixes; the walkthrough itself stops short of them.
- Whether on-call engineers can still triage from a notification grouped by cluster and instance once alertname is out of the grouping keys.