Build1 publisher3 min readPublished
AWS's turn-level metric separates the one broken turn from the three that inherited it
The Agent Evaluation Metric splits correctness into truthfulness and completeness at every turn. That buys attribution to the turn that actually broke, and charges you an expected value for each turn you want to grade.
The Engineer · Build desk

What happened
- AWS has introduced the Agent Evaluation Metric, a decomposable turn-level scheme for measuring agent quality, and correctness is the first dimension it fills in.
- Its worked example has an agent pass profit instead of revenue at turn 2 of a five-turn sales report conversation, with that value silently propagating through turns 3 to 5.
- AWS names three gaps in current practice: task scores do not say which dimension broke, single-turn scores ignore propagation, and holistic scores cannot tell a factual error from a missing required field.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The measurement bill grows with the conversation: ten per-turn judgements stand in for one task-level score on a five-turn trace, and that cost recurs on every eval run rather than once at the end.
- decision Anyone standing up a multi-turn eval now picks between one cheap verdict that cannot locate the fix and a per-turn harness that needs per-turn ground truth to compare against.
- contradiction AWS's own survey concedes some tools already do trace-level root cause analysis, so turn attribution alone is not the new capability, and the named-sub-metric decomposition is the part that has to earn its keep.
- capability If the pattern holds, adding safety or instruction retention becomes a new named sub-metric rather than a rewrite of the evaluation harness.
The failure class that makes this argument work is a successful tool call with a wrong argument. In the AWS example the agent selects the right action for the sales report and passes "profit" where "revenue" was expected [3]. Nothing raises. The trace looks clean, the artifact renders, and the later turns go on refining something correctly shaped and wrong.
So the scoring definition matters more than the block diagram. AWS defines truthfulness as factual consistency between the values the agent produced and what was expected, covering parameter values inside tool calls as well as statements in prose [6]. To compute that at turn 2, the judge needs the expected parameter value for turn 2. That is a different label from the one an outcome score needs, and an eval set built around final answers does not contain it. Completeness is a cheaper check by comparison, because it asks whether required elements are present rather than whether they are right [6]. Underneath both, AWS puts tool and action selection as the structural foundation rather than a third sub-metric [7].
Here is the arithmetic that makes the case, in the currency of turns. The example runs five turns, the error starts at turn 2, and it propagates through turns 3, 4 and 5 [4]. Four turns therefore emit wrong output and exactly one of them needs a change, so three of the four bad turns, 75 percent, are inherited [14]. That ratio is the whole argument for per-turn attribution, and it comes from a hand-built illustration rather than a measured trace population.
The composition step is where I would push. AWS describes sub-metrics measured independently and then recombined into a top-level score [16], and in this material the recombination is named rather than numbered. That gap matters because the post's own complaint against goal completion is that a score of 70 percent does not tell you whether the failures were factual errors, missing information, or wrong tool choices [11]. A composite whose weights you cannot inspect reproduces that opacity one level up. The diagnostic value sits in the per-turn sub-metrics; the single indicator on top is for a dashboard.
What AWS claims as new is the decomposition into named, separately measurable sub-metrics tracked turn by turn, which it says existing tooling does not provide [10], and it says so plainly: the headline contribution is the decomposition itself [8]. In my context that is the right trade when trajectories are long and the tools take parameters, because the thing you end up fixing is one parameter mapping at one turn rather than a conversation. On a two-turn retrieval bot, the outcome score already points at the only turn that could have broken, and the per-turn labels are pure overhead.
What to watch
- Whether a follow-up post publishes the arithmetic and weights that recombine truthfulness and completeness into the single indicator.
- Whether AWS reports judge latency and cost per turn, which is what decides if per-turn scoring runs in CI or nightly.
- Whether safety and instruction retention actually land as added sub-metrics without changes to the evaluation mechanism.