A support agent's faithfulness check passed on documents from 2024 and 2023

Faithfulness checks an answer against the documents the model was handed. Both of those documents came back stale, so the answer was faithful and wrong at the same time: the request named product version 2026.3 ^[12], and the same trace carries faithfulness as pass and requested_version_answered as fail ^[13]. The second verdict is in the record only because somebody wrote that assertion.

Run the counterfactual: had a model.choose_tool span sat between each search, the same slow, stale answer would point at the prompt or the tool descriptions rather than at the retry code in the harness ^[19]. "Same symptom, different file to open," the article says ^[20]. Reading it that way depends on trusting that every model call on the path emits a span.

The coverage number matters here. The 37 points between teams that instrument some services and teams that instrument all of them ^[3] are the teams that cannot separate a retry the model never asked for from a call they do not trace. For an error rate, partial coverage means you see fewer incidents. For a trajectory, it means the evidence you do have can point at the wrong file.

The article puts the three identical searches at 2.6 seconds, from spans of 0.9, 0.8 and 0.9 ^[14][10]. Against a 12.4-second run that is roughly 21%, while the single generate_answer call at 8.1 seconds is roughly 65% ^[15][16]. The listed spans sum to 11.7 seconds and the header says other work is omitted ^[17][9]. On latency the retries are the smaller half of the problem, and the article's other charge is that if the harness appended all three result sets to the context, "you paid for them twice, in latency and in tokens" ^[22].

Sometimes the repeat is warranted. A 200 from a search backend can carry an empty hit list, or every hit under your relevance threshold, and retrying on that is legitimate ^[21].

Reproducing this diagnosis is instrumentation work, per service. Release, retrieval config and feature-flag state have to be attributes set on the root span at span start, not reconstructed later from a deploy log ^[23]. Every model call and tool call has to be a span, in order, with arguments and results, stitched across service boundaries by context propagation ^[24]. Link evaluation results to the trace ^[13]. Taking the question back to the IDE, the article's advice is to bound it: hand the assistant the service, the release, the time window and the trace IDs ^[25].

Dynatrace asked 919 enterprise leaders about coverage ^[1], and coverage is what those percentages describe. The claim that the gap was manageable for deterministic services and becomes a liability with agents ^[4] is an argument about failure modes, illustrated with telemetry the article labels invented and non-standard ^[9]. For the argument to transfer to your service, the harness has to emit spans for the calls it makes on its own initiative. In my view that is the code least likely to be instrumented, because nobody writes tickets against it until an incident makes the retry visible. "A diff is not evidence. It's a statement of intent," the article says ^[26].