Sentry points a nightly Claude routine at its own agent's traces, and files the tickets before standup

Sentry has described a scheduled Claude Routine that reads around 800 of the previous night's Seer conversations before anyone on the team looks at a dashboard, flags the ones that look broken, and files tickets for anything new ^[1]. Seer is Sentry's own AI agent for triaging and fixing errors ^[2], so this is a reasonably clean statement of the underlying position: traces do not pay for themselves until something reads them on a schedule.

The before state is the recognisable part. According to Sentry, checking on the agent's behaviour meant a custom, one-off flow for pulling data out of a hosted Langfuse instance ^[3], where querying options were limited enough that most of the work was hand-rolling a parser for the raw JSON in each trace ^[4]. Someone had to remember to trigger the analysis manually, and automation kept losing to friction ^[5]. The setup also could not send data to the other tools the team used day to day ^[6].

The replacement runs each morning and connects through the Sentry MCP server, which lets Claude query Sentry directly to run Agent Tracing queries against the previous night's conversations ^[7]. The sequence is mechanical: aggregate conversation counts, aggregate tool error counts and rates, sample spans from the errored tools, sample full conversations with and without tool errors to judge whether the agent's final verdict matches its reasoning, pull in the Seer codebase for context, roll it into one report, then search the Linear project and file only what is not already tracked ^[8]. Sentry says the prompt is deliberately plain so the model does the judgment work ^[9], and that the division of labour is MCP for pulling and filtering, Claude for reading, sampling and deciding what deserves a ticket ^[10].

The published example output is the part worth reading. One run covered roughly 445 to 551 conversations and about 11,000 tool calls ^[11], which works out to something near 22 tool calls per conversation ^[1]. It found no verdict-quality problems, and noted the agent correctly downgraded its confidence when it lacked repo access instead of fabricating ^[12]. It did find that about 21 percent of conversations, 83 of 400 sampled, hit at least one tool error, mostly self-corrected retries ^[13]. Two named causes: Search Code being fed a malformed repo_name that appended a monorepo subpath to the real slug, producing "repository not found" until retry ^[14], and a tool receiving a guessed-wrong project_slug such as "mobile" before self-correcting ^[15]. Separately, two conversations showed 30-to-56-second hangs ending in an opaque "internal error", which Sentry calls a real backend latency signal distinct from the guessing pattern ^[16]. All of it went into Linear ticket AIML-123 with example conversation links and error-rate tables ^[17].

Two caveats the post does not resolve. The 400-conversation sample behind the 21 percent figure covers roughly four fifths of that day's traffic, so the rate is a sample, not a census ^[2]. And the example run's 445-to-551 conversations sits well below the roughly 800 in the opening claim, with no reconciliation offered ^[3]; different window, different feature filter, or a quiet night are all plausible and none is stated.

What to watch: whether the tool-error rate actually falls after tickets like AIML-123 land, since a self-correcting retry loop is a cost and latency problem rather than a visible outage; whether the routine's ticket-filing precision holds up, because nothing published measures its false positives or the findings it missed; and what the nightly run costs in tokens against the engineer-hours it displaces.