AWS puts agent evaluation on OpenTelemetry, and on exactly three span roles

The load-bearing sentence in the AWS post is not about frameworks at all. It is the assertion that every major framework supports OpenTelemetry, either natively or through a community instrumentation library ^[11]. If the spans arrive, the scorer runs. If they do not, decoupling evaluation from the SDK buys nothing, and "community instrumentation library" is carrying a lot of weight in that sentence.

The reading surface is deliberately small. AWS says the evaluators need three span roles: an invoke agent span holding the user prompt and final response, inference spans holding message history and model replies, and execute tool spans holding tool name, input parameters and result ^[13]. Set that against the ten OpenInference span kinds the same post enumerates, and seven kinds are collected but never read by the evaluators ^[9]. Retrieval, reranking, guardrail and memory spans are described as context the evaluators do not require ^[4]. That is a defensible boundary, and it also fixes what a score can possibly mean: a retrieval that returned the wrong documents is only visible to the grader if it reached the model's message history ^[13].

Then comes the part that decides whether any of this is portable. Frameworks and instrumentation libraries record those same three roles under different attribute names, different nesting structures and different span-naming conventions ^[5]. So the agnosticism is not in the wire format, which everyone already shares; it is in whatever normalisation layer maps six named frameworks ^[10] onto three roles. AWS's forward-compatibility argument covers the easy failure: an unfamiliar span kind is skipped rather than treated as an error ^[4]. The harder failure is a role that is present but recorded under an attribute the classifier does not expect, which does not announce itself as an error either. It just produces a thinner transcript and a score computed from it.

Worth noting where the neutral standard ends. AgentCore runtime handles hosting, scaling, memory and observability ^[6], and on that runtime the collector is the AWS Distro for OpenTelemetry, routing spans and event records into Amazon CloudWatch ^[12]. Framework choice becomes cheap to change; the telemetry destination named in the post is AWS's own.

For a team already running LangGraph for orchestration or LlamaIndex against a retrieval pipeline ^[3], the question this raises is narrow and answerable: does your instrumentation actually write tool results into the execute tool span, and does the top-level span carry the real final response rather than an orchestration artefact. Those two attributes are what the evaluators grade ^[13]. Everything else in the trace is decoration as far as the score is concerned.