Loading today’s stories
Build1 distinct publisher3 min readUpdated
A dev.to write-up argues per-request capture of prompt, model version, tokens, cost, latency and output quality is now a required line item, because crash-oriented telemetry has no field for any of it.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The failure that matters in an AI feature does not look like a failure. In a dev.to explainer, an engineer who built a retrieval chatbot at Keploy describes the shape precisely: HTTP 200 in 900ms, grammatically perfect prose that happens to be wrong, or that ignored the document you just retrieved for it, or that called the refund tool when the user only asked a question [6][10].
The reason your stack misses this is structural, not a tuning problem. Conventional monitoring is built around status codes, error rates, p99 latency and memory, on the assumption that a broken thing looks broken [5]. By every measure the dashboard has, the service is healthy [7]. Underneath sits an older assumption that nobody writes down: the same input gives the same output, so you replay the request and watch it break again [1]. Send the same request to a model twice and you get two different answers, neither of which threw an error [2].
The author's definition of AI observability is a list of per-request fields: the prompt, the model version, tokens, cost, latency, tool calls, and a judgement of whether the output was any good [3]. Compare that list against the crash-oriented one and the overlap is zero [19]. There is nowhere in standard telemetry to put "this response cost 14 cents", "the model version changed under us last Tuesday", or "the retrieved context was garbage", because those are not infrastructure facts [8].
The mechanics are unglamorous. One question to a retrieval system is a chain, not an operation, and a trace is that chain written down as spans, each with its own start time, end time, inputs and outputs [11]. The question opens a root span; embedding it into a vector is a span with its own model and cost; the vector search is a span whose important content is which five chunks came back; the model call carries model name and version, temperature, prompt and completion tokens, cost in dollars, total latency and time to first token; each tool call is a child span [12].
That structure buys one specific thing: the difference between two fixes. If the search returned five irrelevant chunks, the problem is chunking or embeddings and the model did nothing wrong; if it returned exactly the right documentation and the model still answered from thin air, the problem is the prompt [13]. Without the trace you have "the bot said something dumb", which the author calls the most useless bug report there is [13].
Notably, the piece drops the "three pillars" framing on purpose, arguing that logs, metrics and traces was designed for deterministic systems and has no slot for whether the answer was good [14]. What replaces it is a field list: the final assembled prompt and response as they went over the wire, not the template [15]; model, version and parameters, because providers ship silent updates and you cannot explain last month's regression without knowing which version answered [16]; tokens in and out plus dollar cost attributed to a user or feature [17]; and latency split into total time and time to first token, which feel like different products to a user [18].
Watch the version field first, since silent provider updates are the failure you cannot reconstruct after the fact [16]. Also note the disclosure: the author's team uses Bifrost, an open-source AI gateway from Maxim, and the argument for it is that per-request recording claims can be checked in the code rather than on a marketing page [9].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Traditional monitoring rests on an unwritten assumption that the same input gives the same output: something breaks, you replay the request, you watch it break again, you fix it.
Send the same request to a model twice and you get two different answers, and neither one threw an error.
AI observability is defined as recording what happened inside an AI system on every request: the prompt, the model version, tokens, cost, latency, tool calls, and a judgement of whether the output was any good.
Monitoring tells you the service is up; observability tells you why it answered that way.
Existing monitoring setups watch for crashes using status codes, error rates, p99 latency and memory, all designed around the idea that a broken thing looks broken.
An AI feature failing returns HTTP 200 in 900ms with grammatically perfect prose that happens to be wrong, or that quietly ignored the retrieved document, or that called the refund tool when the user only asked a question.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One first-person explainer, no external corroboration or data
The cluster contains a single dev.to post by one practitioner. Its strongest material is definitional and structural — the span breakdown of a RAG request and the per-request capture list — which is internally coherent and checkable against common practice. But every empirical premise (providers shipping silent model updates, rival tools stopping at a marketing page, the diagnostic payoff of tracing) is asserted rather than demonstrated: no benchmarks, incident records, repository references, or before/after numbers appear. The one arithmetic result, zero overlap between the crash-field list and the AI-field list, holds only over lists the author himself chose.
Two self-reported anecdotes, no measurable uptake
Adoption evidence is limited to first-person disclosure: the author's team uses the Bifrost gateway from Maxim, he built a documentation RAG chatbot at Keploy, and he built an internal per-request logging tool at an unnamed contract role. There are no user counts, download or deployment figures, customer names, pricing signals, or third-party usage data anywhere in the cluster, so nothing supports a claim about how widely per-request AI telemetry is actually instrumented.
Modest claims, mildly overstated by 'required line item' framing
The post is explanatory rather than promotional and its central observation — that crash-oriented telemetry cannot see a fluent-but-wrong 200 OK — is well within what the reasoning supports. Two things push it slightly positive: the necessity framing (per-request prompt, version, cost, latency, quality capture as a required line item) is argued from anecdote with no measured outcome, and the single named tool is credited with line-by-line auditability plus an unverified contrast against rivals' marketing pages. Nothing extraordinary is claimed, so the gap is small rather than severe.
Partial disclosure, one vendor promoted as the worked example
The author states plainly that his team uses Bifrost and that he will use it as the example throughout, which is a real disclosure, and he names his prior employer Keploy. But the post also singles that vendor out for praise while disparaging unnamed competitors' telemetry marketing, and the supplied text says nothing about employment, sponsorship, or affiliation with Maxim beyond 'my team uses it'. That mix — self-published developer platform, one favored named tool, no formal disclosure statement — puts incentive pressure in the middle band rather than at either extreme.
Confident on framing, weak on anything empirical
Confidence is moderate-low overall. The definitional and structural claims can be assessed directly from the text and are unlikely to be wrong in substance, so the assessment of what the post argues is solid. What cannot be assessed from this cluster is whether the argued necessity holds in practice, how widely such instrumentation is deployed, and whether the named gateway records what is claimed — all of which depend on a single, self-interested, uncorroborated account.
leadership
Dynatrace buys Arize for $915M, and AI observability stops being a standalone purchase1 distinct publisher
build
A RAG demo becomes a product at the tenant boundary, not the retriever1 distinct publisher
build
Inference outside the Django app is what makes a multi-tenant RAG support product operable1 distinct publisher
build
Semantic code search over a monorepo is now a plumbing job, and the plumbing is the hard part1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 17, 2026