Build1 publisher3 min readPublished
A demo-to-production LLM checklist prices 50,000 daily calls at two different rates
The failure modes in a dev.to writeup on shipping LLM apps are each ordinary engineering work with an owner. Its cost example implies a per-call price 3.6 to 9 times below the demo figure it starts from.
The Engineer · Build desk
What happened
- A developer writing on dev.to describes shipping an LLM app that passed 10 hand-picked test questions, then sharing it with five real users and watching it fail within the hour.
- The four reported breakages were hallucination on unanticipated questions, a timeout after someone pasted an entire novel, gibberish in reply to a question asked in Hindi, and a user defeating the system prompt with "Ignore all previous instructions."
- The post catalogues the query shapes real users bring instead: vague, compound, mistyped, unanswerable, and in languages the developer never planned for.
- For latency the post says "Research shows that user satisfaction drops dramatically after 3 seconds of wait time. After 10 seconds, most users leave", citing unnamed research.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Anyone sizing a budget from this post has to supply their own model price, because its two estimates for the same traffic sit 3.6 to 9 times apart per call.
- decision Moving the latency target from average to P95 changes which requests get engineering attention: the 5 percent waiting fifteen seconds are the ones the post says walk away.
- capability Input length caps, timeout handling, output filters and a fallback that admits missing information are all schedulable this quarter, with no dependency on a model upgrade.
- exposure Teams that put customer records or internal documents behind a chat box are, on the author's reading, one successful injection away from an incident report.
The cost example is the number a reader will lift. Ten test queries cost about $0.12 [11], which is $0.012 a query [1]. The production case is 10,000 users making five queries a day, or 50,000 API calls [12]. At the demo rate that bills $600 a day, about $18,000 a month [2]. The writeup puts the same traffic at $2,000 to $5,000 a month [12]. Divide that range by 1.5 million monthly calls and you get $0.0013 to $0.0033 per call [3], between 3.6 and 9 times under the demo rate [4].
Token assumptions are where the gap sits. A 4,000-token context plus a 500-token response is 4,500 tokens a call, so 1.5 million calls is 6.75 billion tokens a month [12]. The $2,000 to $5,000 range works out to $0.30 to $0.74 per million tokens blended across input and output [5]. The model behind those numbers is unnamed. For the estimate to transfer, your blended rate has to land in that band; on pricing like the demo figure implies, multiply by four to nine. The spend controls listed are aggressive caching, semantic caching for similar queries, per-user rate limits, daily spend monitoring instead of monthly, and routing simple questions to cheaper models [13].
P95 is the instruction that needs no external support to stand up. An average of three seconds can hide 5 percent of users waiting fifteen [10], and those requests are invisible to any dashboard that reports a mean.
Injection advice comes down to a list of ordinary controls: keep user input out of the system prompt, sanitize input, filter output, run known injection attacks against the app, and return a fallback response when the model's output looks suspicious [7]. Each of those is a ticket with an acceptance test. The post argues that where an app handles customer information, financial records or internal documents, a successful injection is a security incident [8].
Retrieval is the most specific section, and the post claims production is "80% edge cases" [16]. A 200-page PDF crashes the chunker with a memory error, embeddings go stale against a document updated yesterday, and two documents contradict each other so the model picks the wrong one [14]. The prescribed handling is input length limits, timeout handling, logging every failure, and graceful degradation: say "I don't have enough information" instead of hallucinating [15].
The scope here is narrow. One developer's experience post, ten hand-picked demo queries and five users [2][3], and the supplied text breaks off inside the sixth item, so three of the promised nine are not in it [6]. The recommended evaluation set of 100 or more queries in real user phrasing [6] is asserted without measurement. What holds is the taxonomy: every failure it names has a fix a team can schedule without touching the model.
What to watch
- Whether the remaining three of the nine items, once published, are engineering tasks with owners or complaints about model quality.
- Measured P95 latency and monthly spend from a production retrieval app at comparable traffic, which would test the $2,000-$5,000 range against something other than an estimate.
- Whether the instruction to test with known prompt injection attacks gets pinned to a named, versioned attack corpus that a CI job can run.