Green dashboards, invented refund policy: the case for a separate AI eval layer

An engineer writing at devops.com describes an AI support agent that had been live for three weeks when a customer screenshot arrived in Slack: the bot had invented a refund policy for a product the company had never sold, made up the SKU and the rules, and returned the answer confidently in 1.2 seconds ^[2]. Latency was green, error rate was green, and by that account every SRE metric said the system was healthy while it was misinforming customers at scale ^[2].

The reason matters more than the anecdote. Uptime, latency and error rates do not measure whether a model's answers are correct or grounded ^[1]. The SRE playbook held for fifteen years because systems were deterministic, same input, same output, and LLM systems break that contract ^[15]. A vendor can push a new model checkpoint silently and the agent develops a new personality; a re-indexed retrieval store can route "what's your return policy?" to a marketing blog post instead of the policy document; neither event registers as a 4xx ^[7]. There is nothing in the monitored signal set that overlaps with the failure ^[16].

The prescription in the piece is evaluation in three explicit places: before release, in CI, and on sampled live production traffic ^[3]. The author's assessment of the field is that most teams have something resembling the first, almost nobody has the CI gate, and the third is where the Slack screenshots live; asked to pick one, they would start with production sampling because it catches the failures you did not predict ^[9]. The alternative, described bluntly, is the vibes check: a PM tries six prompts, says it feels good, and ships, with no regression suite, no quality baseline and no rubric ^[8].

Cost discipline is what makes this survivable. The recommended stack is layered: deterministic code checks run on every trace for free, LLM judges and other sampled evaluators run on a subset because judges cost real money and will consume the budget if run inline, and human domain review sits at the top, slowest and most valuable ^[4][5].

The eval set itself is supposed to come from your own incidents, not a vendor. Generic benchmarks score 95 percent on your system while testing nothing customers care about ^[13]. The loop described is: pull production traces weighted toward anomalies, low-confidence scores and complaints; have one person label them, on the argument that consistency beats coverage and two labelers create noise that looks like signal; cluster failure modes, hallucination, tone, wrong tool call, and give each its own judge; then promote the failures into permanent regression tests so the same defect cannot recur silently ^[6][10]. Where data is thin, the advice is to synthesise inputs only and run the real application on them, because letting a model generate both sides yields a set that scores 99 percent and means nothing ^[11].

For retrieval-augmented systems the ordering is explicit: context precision and recall first, since nothing downstream matters if retrieval is broken, then faithfulness and answer relevance, then diagnostics such as citation accuracy, noise sensitivity and per-chunk hallucination rate, which the author says belong in investigations rather than alerts ^[12]. Guardrails and evaluators are told to stay in separate code paths and separate boxes on the architecture diagram ^[14].

What to watch in your own stack: whether a quality gate exists in CI at all, who owns the label set, and whether last quarter's worst production output is now a test that fails when it recurs ^[3][6][10].