Your LLM App Returns HTTP 200 While It Fails, and finish_reason Is the Field That Tells You

A practitioner guide published on devops.com argues that an LLM-powered application can be up, serving traffic and returning HTTP 200 responses while failing badly, producing hallucinated content, silently truncated outputs, unsafe responses or degraded quality after a provider quietly updates the underlying checkpoint ^[1]. That matters because standard infrastructure monitoring reports none of it, so the incident starts when a user complains rather than when a chart moves ^[1][2].

The mechanical reason is a type problem. Conventional APM tracks latency, error rates and throughput, which the author calls necessary but not sufficient for LLM systems ^[3]. A conventional API returns a well-typed response or throws an exception; an LLM returns a string, and that string may be correct, plausible but wrong, in a format that breaks a downstream parser, in violation of content policy, or truncated because the context window was silently exceeded ^[4]. None of those register as an error in standard monitoring ^[4]. The failure modes that matter most, per the guide, are semantic rather than structural ^[5].

Hence the recommended log line: model, workflow, token usage, latency, time to first token, finish reason, cost and quality signals ^[6]. That is eight fields, and only one of them separates a completed answer from a guillotined one ^[7]. The guide singles out finish reasons precisely because truncated responses can still appear as successful API calls ^[8]. If your dashboard aggregates 200s and p95 latency but drops finish_reason on the floor, a context-window overflow is indistinguishable from a good day.

The guide sorts production failures into four classes, each needing a different instrumentation approach: quality drift, prompt failures, cost anomalies and latency degradation ^[9][10]. Quality drift is attributed to three causes: the provider updated the model, the prompt met input distributions it was not tested against, or a downstream change altered the context reaching the model ^[11]. Detection is a baseline problem, not a logging problem: representative inputs with expected outputs or rubrics, run on a schedule against samples of production traffic, with the trend line rather than any single run as the signal ^[12].

Prompt failures are harder because they present as partial successes, output that is wrong in ways that need domain knowledge to spot ^[13], and because pattern-based failures hit a narrow slice of inputs and disappear in aggregate metrics ^[14]. The proposed instrumentation is full prompt-response pairs with PII handling, structured metadata tagging the result type, and enough sampled traffic for human review ^[15]. Cost anomalies are the most self-inflicted class: token-based billing plus a bug that attaches a large context document to every request, a prompt template that grew, or a change that turns single-turn calls into multi-turn conversations ^[16].

The cost-aware answer to evaluation is the useful bit for anyone budgeting this work: rule-based checks on every response, model-graded evaluation on a sample only, which the guide says catches quality regressions without doubling inference costs ^[17]. This is one operator's account, drawn from two years running a production LLM application handling tens of thousands of requests daily ^[18], not a benchmarked study.

Watch whether finish_reason is a first-class dimension in your own dashboards, or a string buried in a payload nobody queries. The excerpt available here breaks off inside the cost section, before latency degradation gets its treatment ^[19], and time to first token is the field that class will rest on ^[6].