Build1 publisher2 min readPublished
A bare PromQL comparison left a stopped-task alert live only between zero and one
A rule can evaluate cleanly, appear in the rule list, report healthy, and still be incapable of going red. One team found four of those in its own stack, and the PromQL one turned on a missing keyword.
The Engineer · Build desk
What happened
- A week-long audit of one team's own alerting found four rules that could not produce a red signal in the exact circumstance each was written for, and every dashboard stayed green throughout.
- The maintenance-task rule compared a vector against a scalar without bool, so PromQL treated increase(...) < 1 as a filter and returned the computed value for every matching series.
- A task that stopped entirely produced increase == 0, which passes the < 1 filter and then fails the Grafana threshold of gt [0], because 0 > 0 is false.
- A second alert watched a Celery failure counter incremented in the worker process while /metrics was served by the web process, two prometheus_client registries with no shared state.
- The trap surfaced because the rule's author mutated their own working < bool 1 back to the naive < 1 to test the linter, and 196 tests passed.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Under noDataState: OK, a rule whose metric pipeline is broken and a rule watching a healthy quiet system deliver the same green, so a filtering expression cannot tell an operator which of the two they are looking at.
- decision Anyone alerting on absence through Grafana has to settle what the expression returns before choosing the threshold, because one keyword decides whether the engine receives a truth value or a measurement.
- exposure A satisfiability check that understands one expression shape and skips the rest reports the same clean run as one that verified every rule, so its output cannot be read as coverage.
- capability Mutating a rule you already know to be correct tests the harness instead of the code, and the post puts each of these shape checks at about ten minutes of work.
Two lines against the engine make it concrete. `vector(0) < 1` returns one series with the value 0, and a Grafana threshold of `gt [0]` on 0 is false [5][8]. `vector(0) < bool 1` returns one series with the value 1, and `gt [0]` on 1 is true [8]. The `bool` modifier turns the filter into the 1/0 comparison everyone assumed they were writing [9]. The live range of the naive version is the open interval (0, 1), so what it reliably catches is a task that has half stopped [7].
The audit's question was not whether anything was alerting. It was "would this alert look any different if the thing it watches had failed?" [1]
Both halves are needed to kill the rule. PromQL passes the computed number down, and the Grafana reduce-and-threshold step then asks whether that number is above zero [21]. For the shape to be dangerous in another stack, both have to be present: a vector-to-scalar comparison written without `bool`, and an evaluation step that treats 0 as not firing. An engine that fires on the mere existence of a returned series reads the same expression differently.
A longer chain runs behind the failure counter. `celery_tasks_total.labels(task=name, status="failure").inc()` had been in the repo since the start, an alert watched it, and there had never been a task failure it should have caught [13]. Past the split registries, Prometheus was not scraping Celery at all: seven targets configured, none of them the worker [15]. The alert's query returned zero series, forever [16].
Grafana's rules run with `noDataState: OK`, which the post calls deliberate and correct for filtering expressions, because a healthy system genuinely produces no rows [16]. The post writes the general case down: "An exporter that is scraped but has stopped receiving events looks exactly like a quiet system." [17]
Four other mutations in the same run went correctly red: a dropped scrape job, a dropped deploy step, a drifted subquery step, a bare staleness comparison [11]. The harness detected four of the five injected faults [20]. The one that escaped was the absence check, because the repo's satisfiability check recognised the `== N` shape and skipped silently on everything else [12].
The fix was not to move the counter. The team stopped asking the application to report on itself and read the broker's own event stream with a dedicated exporter, starting the worker with `-E` [18]. That opened the next trap: an exporter pointed at a worker without `-E` still emits worker-liveness metrics and zero task metrics [19].
What to watch
- Whether the repo's satisfiability check is extended past the == N shape to comparison expressions generally.
- Whether the new broker-event exporter gets an alert that separates "worker up, events off" from a quiet queue.
- The published post breaks off mid-sentence on the -E trap, so the full write-up of that case is still to come.