Build1 publisher2 min readPublished
A poorly scoped supervisor prompt sends 20 percent of requests to the wrong specialist
An AWS post on agent monitoring describes failures that return clean responses and throw no exceptions, and answers them with a judge model that scores live interactions for helpfulness, correctness and goal completion.
The Engineer · Build desk
What happened
- AWS says a supervisor agent with a poorly scoped prompt does not raise error rates and instead starts routing 20 percent of requests to the unintended specialist while infrastructure metrics stay green.
- In a second example, a missing IAM permission on an agent's execution role leaves it unable to invoke its foundation model, so it returns an empty response without throwing a 500 error.
- The write-up is built on an airline reservation system AWS assembled itself, with four specialized agents, pairing AgentCore Evaluations for quality scoring with AWS DevOps Agent for incident investigation.
- AWS DevOps Agent handles the infrastructure side, analyzing CloudWatch logs, tracing failures across service boundaries and correlating IAM policies with invocation logs and orchestration traces.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Continuous scoring bills model inference to grade model inference, and with no sampling rate or per-evaluation price in the post, the buyer sizes that spend from their own traffic.
- constraint A swarm has no fixed execution graph, so a team cannot pre-place assertions at the handoff points and has to check outcomes after the fact instead.
- decision A goal-completion score only gates a release once someone sets the number it must clear and owns the call when it drops, and that threshold is not something the framework supplies.
The dashboard stays green because of where the failure sits in the chain. AWS gives the case of a booking agent that stops completing reservations while the logs still show successful tool executions, because the problem happened three calls deep in a chain that never surfaced an exception [3]. From the infrastructure's side, each call went out and each call came back.
AWS draws the line between execution and effectiveness. CloudWatch metrics show whether systems executed correctly, but not whether agents helped users accomplish their goals [4]. AgentCore runtime already emits monitoring data to CloudWatch through OpenTelemetry instrumentation [5], so the telemetry arrives. No trace carries an assertion about whether the answer was the right one.
AgentCore Evaluations is meant to supply that assertion. The post describes it as a quality assessment framework that continuously scores live interactions using LLM-as-a-Judge, with metrics on helpfulness, correctness, goal completion and other quality dimensions [6]. A second model reads the interaction and grades it. For a misroute to appear as a score drop, the grade has to be anchored to what the user asked for, because a wrong specialist that answers fluently and completes its own task will pass a check on fluency.
At a thousand requests a day, 20 percent is 200 conversations answered by the wrong specialist with the error count unchanged [12]. AWS offers that share as an example of the failure class [1]. The post does not give a sampling rate, a cost per evaluation, an added-latency figure, or any measure of how often the judge agrees with a human label [13]. Whether continuous scoring is affordable on production traffic depends on the first two of those.
The other layer in the post is a tool AWS describes as one that "acts as an on-call engineer for your infrastructure" [9].
The routing pattern makes the quality side awkward to wire up in advance. In the Swarm pattern a supervisor routes work dynamically to specialists based on the task, so there is typically no fixed execution graph to instrument, and failures can occur at multiple handoff points and propagate unpredictably [10]. Strands Agents, the open source SDK in this stack, supports Swarm, Graph and Agents-as-Tools [11], so the pattern itself travels between frameworks. The scoring framework is described as integrated into AgentCore runtime [6], so a team that takes the quality layer takes the runtime with it.
What to watch
- Whether AWS publishes judge-to-human agreement rates or a per-evaluation price for AgentCore Evaluations.
- Whether the scoring framework becomes usable outside AgentCore runtime, which the post ties it to.
- Whether the airline reservation demo is followed by a customer workload with measured misroute rates.