Build1 publisher2 min readPublished
Strands Evals scores skill selection and invocation as pass/fail, rates instruction following on a five-level scale
AWS has added three skill-focused evaluators to Strands Evals and Bedrock AgentCore Evaluations. Two of them call a model once per invoked skill, and the third is a deterministic check that exists only in Strands.
The Engineer · Build desk
What happened
- The Strands Evals SDK and Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, have added evaluators aimed at the skills an agent loads rather than at its final answer.
- A skill is a reusable set of instructions, usually kept in a SKILL.md file, that teaches an agent one domain task such as redacting a contract or following a team's pull-request conventions.
- AWS names two failure modes that general output-quality metrics can miss: the agent invokes an unsuitable skill, or invokes the right one and skips part of its instructions.
- Skill Selection Accuracy judges whether each invoked skill was an appropriate choice for the task and returns a binary result for every skill the agent invoked.
- Skill Invoked is a deterministic check of whether a named skill loaded successfully; it runs without a model call, and AWS describes it as specific to Strands Evals.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Grading skill adherence is itself an inference workload, billed per invoked skill per case, so whoever owns the CI budget now owns a recurring model spend that grows with how many skills a run touches.
- constraint The evaluators read a recorded trajectory or an OpenTelemetry trace, so a team without agent tracing in place has to instrument first and cannot start with the test suite.
- decision Teams that chose the open skill format for harness portability have to decide whether a merge gate may depend on a routing check that ships in one SDK.
- capability Domain guidance was already editable without touching the model; the new part is that the edit can be checked in the same recorded run that produced the bad answer.
A run in which the agent loads nothing still has to fail somewhere, and the failure has to come from the deterministic check. Skill Selection Accuracy needs an invoked skill before it has anything to judge [4]. That check is the one you assert by name in a test suite, one line per skill you expect the router to reach [6][11].
Two of the three evaluators call a model. Skill Instruction Following returns a five-level rating, grounded in evidence, for each prescribed step of the skill it is grading [5]. One test case in which the agent invokes two skills therefore produces four model-graded judgments: a selection result and an instruction-following rating for each invoked skill [1].
A skill packages its instructions together with tool bindings, meaning the APIs, Model Context Protocol servers or local commands it depends on [3]. A successful tool call only tells you the API answered. In the HR assistant example AWS walks through, the agent holds one skill for paid time off planning and one for employee benefits. An employee asks about dental and vision benefits. If the tool call succeeded but the agent chose the wrong playbook, Skill Selection Accuracy isolates that routing decision [12].
A selection failure and an adherence failure lead to different edits. One points at the catalog and at how each skill describes itself to whatever does the routing. The other points at the steps written inside the file. Either failure can produce a fluent, plausible response that leaves your domain knowledge out of it [8].
AWS publishes the prompt template and the rubric for both model-based evaluators [7]. Read those before a five-level rating becomes a merge gate. The rating is a judgement about evidence in a trace, made by a model against a rubric someone else wrote. The post does not report how closely those graders agree with a human reviewing the same trajectory [15].
The suite is still one suite. The skill format changes the granularity of the assertion: each invoked skill inside a run now carries its own score. AWS's own framing is that you read the per-skill results to choose the right fix [11].
What to watch
- Whether AgentCore Evaluations gains a deterministic SkillInvoked equivalent of its own.
- Whether AWS publishes agreement data between the five-level Skill Instruction Following rating and human review of the same trajectories.
- Whether the published rubric changes how teams write SKILL.md files, for example enumerated steps written to be graded.