An empty Detected fields panel is a Loki capacity incident that has not billed yet

A field write-up published on dev.to describes a client edge cluster where `{job="nginx"} |= "500"` in Grafana Explore returned every 500 error you would expect, while `{job="nginx", status="500"}` came back empty ^[1][2]. The author's framing is the useful part: nobody noticed the gap until the Loki ingesters started OOMing ^[3].

Treat the two symptoms as one incident at different stages. Stage one is cosmetic: the raw nginx line renders fine in Explore, but the Detected fields panel is empty, with no `remote_addr` and no `status` ^[4]. Stage two is billable: memory spikes on the ingesters, plus `too many streams` or `per-stream rate limit exceeded`, which the author says usually arrive right after someone "fixes" the regex by promoting every field to a label ^[6]. That fix looks like a success in isolation, because fields finally extract, while introducing a cardinality explosion ^[7]. So you get a quiet period of unparsed ingest, then a loud period of over-labelled ingest, and only the second one pages anyone.

What makes the quiet period long is the absence of any failure signal. Promtail may log `pipeline stage failed to parse` at warn level, or it may log nothing at all and ingest the lines unparsed ^[5]. The regex compiles, Promtail starts, and the config-load step raises nothing; the pattern simply does not match ^[9]. Two RE2 specifics widen the gap. Promtail and Grafana Alloy use Go's RE2, which has no backreferences and no lookahead or lookbehind, so a pattern validated on regex101 in PCRE mode can misbehave silently ^[10], and a greedy `.*` inside a quoted field will swallow the next quoted field ^[11]. Named capture groups are mandatory: `(?P<remote_addr>\S+)` produces a usable field, while a bare capture group is discarded by the regex stage with no warning ^[12]. YAML escaping of literal brackets and quotes is the other trap, and a malformed escape does not throw a config error, it breaks the match at runtime ^[13]. Config load, process start, and the log stream can therefore all look clean while nothing is being extracted ^[21].

The stated root cause is mundane: a regex copied from a tutorial that assumed nginx's default combined format, running against a server with a custom `log_format main` that has a different field order, extra fields, or different quoting ^[8]. The remedies are equally mundane. Pull the exact directive out of `/etc/nginx/nginx.conf` and build the pattern field by field against it ^[14], then validate offline with `promtail -dry-run -inspect` against a real sample line before touching the running pipeline ^[15]; regex101 in Golang mode is the nearest approximation if the binary is not to hand ^[16]. The breakages the author sees most are unescaped brackets around `$time_local` and unescaped quotes around `$request` and `$http_user_agent` ^[17]. Order the stages regex, timestamp, labels, drop, because labels placed before regex has no captured data to reference and extracts nothing ^[18] - a mistake the author reports shipping to production twice on the same team ^[19]. Wire the timestamp stage to `time_local` with the Go layout `02/Jan/2006:15:04:05 -0700` so Loki indexes request time rather than scrape time ^[20].

Worth watching: whether your alerting can see stage one at all. The write-up offers no interval between the broken stage and the memory spike beyond "some time later" ^[6], which is another way of saying the detection budget is unknown. An empty Detected fields panel and a rising unparsed-line volume are the cheap signals; stream count after any regex change is the one that decides whether the fix cost you more than the bug.