Build1 publisher3 min readPublished Updated
Refunding a customer's $2,300 order emits the same trace as refunding their $40 one
LangChain's survey of 1,340 practitioners found 89% had observability on their agents and 37.3% ran online evaluations. The OpenTelemetry GenAI attribute registry explains why only the second number measures quality.
The Engineer · Build desk
What happened
- LangChain's State of Agent Engineering report, published 23 May 2026 from 1,340 responses gathered between 18 November and 2 December 2025, found 89% of practitioners had implemented observability for their agents.
- The same survey named quality, meaning accuracy, relevance, consistency and tone, as the top barrier to getting agents into production, cited by about a third of respondents.
- The OpenTelemetry GenAI semantic conventions cover model, provider, token counts, finish reason and operation name, and hold no correctness field unless an evaluator writes one in.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Alerts built on latency percentiles, token spend or error rate have no trigger for this failure class, because the wrong run and the right run land on the same point of every one of those series.
- decision Buying an eval product does not close the gap on its own. Someone on the team has to write down what a correct refund is before any judge can score one.
- exposure Write-capable tools put money inside the unmeasured region: the agent moved $2,260 more than the complaint warranted and the trace reported success.
- contradiction The survey reports what practitioners say they have installed, so the category-error diagnosis has to stand on the OTel attribute list, not on outcome data from the 1,340 responses.
When the agent calls `issue_refund`, the payments API returns success and the span closes with status `OK`. In both sample runs the dev.to post prints, all three spans close `OK` and `error_count` is zero [9][10]. The post wrote: "The span is green because the API call succeeded, and the API call succeeding is the only thing a span status has ever meant." [12]
Run B took 3,204 ms against Run A's 3,180, a difference of 24 ms, or 0.75% [9][10][16]. It read four more input tokens and emitted five fewer [22]. Both stopped with a `stop` finish reason [9][10]. The difference sits outside the telemetry: in Run B the customer had a $40 order they were complaining about and a $2,300 order they were not, and the refund went to the second [11]. That is $2,260 more than the complaint asked for [17]. On a latency percentile chart, a spend chart, or an error rate, the two runs are the same point twice [21].
The reason nothing separates them is the attribute registry. The OpenTelemetry GenAI semantic conventions define which model answered and which provider served it, input, output, cache-read, cache-creation and reasoning token counts, the finish reason, and an operation name of `chat`, `tool_call` or `agent_run` [13]. Per the post, there is no attribute for correctness until you run an evaluator and write one in [8]. That boundary is the right one. A cross-vendor standard cannot encode your refund policy, and a span status that claimed to certify the business outcome as correct would be unsettable by the HTTP client that sets it. The post's explanation for the adoption asymmetry is procurement: tracing is a drop-in purchase, and evaluation requires someone on your team to state what correct means for the domain [14].
The adoption figures are what 1,340 practitioners said about their own stacks between 18 November and 2 December 2025, published on 23 May 2026, 172 days after the collection window closed [5][18]. For 89% and 37.3% to describe your team [1][3], "implemented observability" has to mean what you mean by it, and "online evaluation" has to cover whatever you actually run against live traffic. Among teams with agents in production, observability was 94% [2].
The two eval numbers do not nest. Offline evaluation was 52.4% and online 37.3%, and fewer than a third of respondents ran both [3][4]. Taking a third as 33.3%, at least four points of the online group have no offline suite [19]. Those teams are scoring production traffic without a fixed regression set to compare it against.
The survey does not link eval coverage to incident rates, and the two traces in the post are constructed illustrations [9][10]. So the argument that the 51.7-point spread between observability and online evals is a category error [15][20] rests on the attribute list rather than on the response data [8]. In my view that is the sturdier footing, because you can check the list against your own exporter in an afternoon.
What to watch
- An evaluation or correctness attribute group added to the OpenTelemetry GenAI conventions would move this measurement into the standard instead of leaving it to each team.
- The next wave of the LangChain survey, and whether online evaluation adoption moves off 37.3% while observability stays near universal.
- Whether eval vendors ship default scorers that work without the customer first defining correct for its own domain.