Grafana wants agent quality tracked with SLOs and error budgets
Grafana Labs proposes turning agent evaluations into SLIs, with a 95% fulfilment target over 30 days leaving a 5% error budget, InfoQ reports. Grafana argues latency and token metrics cannot show an answer was wrong, and budget burn exposes slow drift.
The Engineer · Build desk
Three figures from Grafana's example SLO. Target: 95% of conversations judged as fulfilled. Error budget at a 95% target: 5 percentage points. Time window of the objective: 30 days.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Conversations to be judged fulfilled, SLO target | 95% | 6 | |
| Error budget at 95% target | 5 pct points | 14 | |
| Objective window | 30 days | 10 |
What happened
- Each evaluation checks one interaction against one behavioural criterion, such as grounding in context, request fulfilment, toxicity, personal-data exposure or prompt injection.
- Grafana keeps these as separate measurements with their own objectives, because an agent can be well grounded but expensive, or accurate but slow.
- The budget doubles as a change policy: while it lasts, teams can try new prompts and models, and as it drains the priority moves to quality work.
- Grafana's route to a target is to measure a baseline with evaluations, let the history suggest a number, and tighten it as the agent matures.
Why it matters
- Who has to decide decision Each team has to pick its own number, because there is little consensus on whether fulfilment or groundedness should sit at 85%, 95% or 99%.
- What it limits constraint The budget counts evaluator verdicts, so a noisy LLM judge spends budget on failures the agent did not commit.
- What it costs cost Every LLM-judged evaluation adds a model call to each interaction it grades, and the team running the agent pays for it.
A single poor interaction barely registers against a 30-day objective, Grafana argues, but a persistent decline steadily consumes the budget [10]. The budget is the gap between the target and 100% [8]. At a 95% target that gap is 5 percentage points [14].
Drift is the case this is built for. An agent can keep responding successfully while its answers get less useful [9]. A model change, a prompt change, a retrieval change or a shift in user behaviour might each cost a fraction of a percentage point a day [9]. Latency, error-rate and token metrics can report a fast, error-free response while that happens [2]. A conventional alert watches for a spike in failed requests [9]. Budget burn watches the aggregate of the verdicts.
Not every check needs a model. Grafana allows deterministic checks such as regular expressions where the behaviour can be identified without an LLM [4].
The 95% in the example is an illustration. InfoQ's report does not include results from Grafana's own deployment, though the practice grew out of monitoring its Grafana Assistant [13]. For the number to transfer, an operator's evaluator would have to agree with its users about what "fulfilled" means. Traffic would also have to be steady enough that 30 days of verdicts says something.
In our view, setting the target from a measured baseline is the sound order, because 95% is a placeholder until a team has its own history [12].
What to watch
- Whether Grafana publishes baseline results and the targets its tooling recommended for Grafana Assistant.
- Whether teams report how closely LLM-judge verdicts match human review, since those verdicts feed the budget.
- Whether the 30-day window holds up for agents with low or uneven traffic.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence35
- Adoption
- Not scored
- Hype gap+5
- Incentives
- Not scored
- Confidence65
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Grafana Labs describes using evaluations, SLOs and error budgets to turn behaviours such as hallucination, task fulfilment, toxicity and cost into measurable targets that can be monitored over time.
- [2]
Traditional metrics can show that an agent responded quickly, hit no technical errors and consumed a certain number of tokens, but none necessarily indicate whether its answer was correct, grounded in available information or useful.
- [3]
An evaluation examines an agent interaction and determines whether it met a behavioural criterion; Grafana's examples include whether a response was grounded in its supplied context, fulfilled the user's request, contained toxic content, exposed personal information, or succumbed to prompt injection.
- [4]
Evaluations can use another language model as a judge, but deterministic checks such as regular expressions can also be used where the behaviour can be identified without an LLM.
- [5]
Grafana argues that agent quality is multidimensional (an agent might be well grounded but expensive, helpful but occasionally toxic, or accurate but slow), so it treats these as separate measurements with different objectives instead of one aggregate agent quality score.
- [6]
Grafana gives the example of an SLO requiring 95% of conversations to be judged as fulfilled over 30 days; the remaining 5% becomes the error budget.
- [7]
When sufficient error budget remains, teams can experiment with prompts, models or agent behaviour; as the budget is consumed, the priority can shift towards improving quality rather than introducing additional changes.
- [8]
Grafana's existing SLO tooling defines an SLO through an SLI, a target and a time window, with the error budget calculated from the gap between the target and 100%.
- [9]
A conventional alert might identify a sudden spike in failed requests, but an agent can continue responding successfully while its answers become subtly less useful; a model change, prompt modification, retrieval change or shift in user behaviour might cause quality to deteriorate by a fraction of a percentage point each day.
- [10]
Grafana argues that monitoring error-budget burn can make drift visible: a single poor interaction may not matter against a 30-day objective, but a persistent decline can steadily consume the available budget.
- [11]
Unlike availability targets, where organisations have decades of operational experience, there is considerably less consensus on whether an agent should achieve 85%, 95% or 99% fulfilment or groundedness.
- [12]
Grafana's approach is to establish a baseline through evaluations, use historical results to help recommend an appropriate target, and tighten the target as the agent matures.
- [13]
Grafana's own Agent Observability work grew from the problem of monitoring its Grafana Assistant.
- [14]
At a 95% target the error budget is 5 percentage points.
- [15]
Because the SLI is built from evaluator verdicts and the evaluator may be a language model, a judge's misclassifications count toward the error budget as if they were agent behaviour.
- [16]
Each LLM-judged evaluation is an additional model call on top of the interaction it grades, so judging adds inference cost per evaluated interaction.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- infoq.comGrafana Applies SLOs and Error Budgets to AI Agent Behaviour
1 article · October 16, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Entities
- InfoQFollow
- Grafana AssistantFollow
- Grafana LabsFollow