Topic
Practices for managing LLM inference cost and latency, including prompt-context trimming, caching, and routing tasks to cheaper models based on difficulty.
AWS puts prompt caching's saving at up to 90 percent on cache hits; its own ten-question example nets about 75 percent, and only while every request lands inside the five-minute default TTL.
Reality
LiteLLM's own Terminal-Bench run puts a gpt-5.4-mini classifier routing across Haiku, Sonnet and Opus at $14.34 against Opus-5's $19.74 for the same 16 solved tasks. The saving works out at 26 cents a task.
Reality
The San Francisco startup records production work and expert corrections, turns them into task-specific tests, and reroutes traffic only when a cheaper candidate passes; the studies behind that are its own.
Reality
Anthropic's prompt caching docs describe an automatic mode that keeps the breakpoint on the last cacheable block of every request. The five-minute clock starts when the request starts, so generation time counts against it.
Reality
GitHub reports cost cuts of 67% on TerminalBench 2.1 and 36% on DeepSWE, all measured by GitHub, while the preview as described hands platform teams no way to set the routing policy or see which model wrote what.
Reality
The Copilot team put a token-shortening utility through its agentic benchmarks, watched task cost rise even as individual responses shrank, and shipped a compressor that only touches install, build, test and progress output.
Reality
AWS, NVIDIA and Heidi report cutting production speech-recognition inference cost by 75% by sharing one GPU through CUDA MPS instead of running a single model instance. How much of that saving transfers depends on the latency envelope you accept.
Reality
The router picks a cheaper model for work that resembles work it has already seen, and falls back to the strongest one when nothing matches. The same traces that enable that call are the proposed training set.
Reality
The two-week Enzyme removal is credible enough, but it is priced against a five-year staffing plan written by people who did not want the project, and Airbnb's published numbers let you check the rate that plan assumed.
Reality
AWS, NVIDIA and Heidi Health report holding sub-second transcription while cutting 16 GPU instances to four. Per-GPU throughput rose only about 1.5x, so the rest of that saving came out of provisioning headroom.
Reality
Auto now picks the model for each task in every Replit account, and getting a specific model back means a Core or Pro plan plus a mode that can charge usage credits. Stripe's reported $8 billion for OpenRouter prices the same layer.
Reality
Scenematic's out-of-distribution gate sends its least understood prompts straight to the expensive render. The logic holds up; the harness meant to prove it still simulates the scores.
Reality
A dev.to hands-on replaces the classifier Lambda with a Step Functions Choice state. The trade is a second Bedrock invocation on every question, including the cheap ones.
Reality
A dev.to writeup documents why cache_read_input_tokens falls to zero mid-run: a cache_control breakpoint walks back at most 20 content blocks, and parallel tool calls clear that in one turn.
Reality
Upstage is selling tool-calling discipline rather than reasoning, and says 370 billion tokens moved through OpenRouter in its first week. The price, the part that matters most, is still qualitative.
Reality
IBM Research ran self-mined guidelines across eight models on AppWorld. One model gained 16.1 points for 5 percent more tokens; another gained nothing at all.
Reality
A new beta fingerprints each request and names the first structural divergence from a prior response id. Until now the only signal was cache_read_input_tokens going to zero.
Reality
A payments company is paying model-lab money for the layer that decides which model gets the request. The routing decision and the settlement decision are converging into one stack.
Reality
GenRec beat a heavily tuned production recommender by 0.006 percent on 10 percent of traffic for four weeks. The online gain is trivial; the labeling economics are not.
Reality