Build1 publisher2 min readPublished
A SHA-256 of the normalized request body decides whether the model gets called
A response-caching walkthrough in The New Stack puts model settings and upstream data inside the cache key, so a version bump flushes the store by design. The semantic tier layered on top is where wrong answers enter.
The Engineer · Build desk
What happened
- A response-caching walkthrough in The New Stack fingerprints the request, its context, the model settings and the upstream data into one exact-match key, and returns the stored answer without calling the model.
- Tier one normalizes the request body, hashes it with SHA-256 and looks the hash up in an in-memory store such as Redis, returning the answer without waiting for inference.
- Tier three checks the exact store first, runs semantic search only on a miss, and promotes a close-enough semantic match back into the exact store under the hash of the new query.
- The author traces the technique to nightly pipeline jobs that recompute aggregations whose inputs never changed, pass every check and ship results while burning compute.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Keeping model settings inside the key means an upgrade or a temperature change misses every stored entry at once, so the saving restarts from zero whenever the model moves.
- cost Provider prompt caching discounts eligible cache reads but keeps charging for output generation, so output tokens stop only when the call itself does not happen.
- exposure Promotion makes a bad semantic match durable: the next identical query is answered from the exact store by hash, with no similarity comparison left to catch it.
- decision Teams whose LLM traffic is bounded and predictable can stop at tier one; free-text user questions push the decision onto the semantic tier and its threshold tuning.
Most of the work goes into normalization. The key covers four things: the request body, the context assembled into it, the model settings, and the upstream data the answer depends on [1]. A request body normalizes cleanly. Upstream data means you need a version or a content hash for every table, document or index the prompt reads, and you need it at lookup time, before you can call a stored answer valid [1][13].
That makes invalidation explicit. Change the sampling temperature or bump the model version, and every key changes with it, so the entire store misses [1]. A cache that survives a model swap is crediting the new model with an answer the old one produced.
Provider prompt caching is a different bill. Providers reuse cached prompt computation and charge eligible cache reads at reduced rates, and output generation stays billable [5]. A response-cache hit never reaches the provider, so no output tokens are generated at all [15][3].
The New Stack piece offers a 0.90 to 0.95 cosine window as a starting point, says the right value depends on your embedding model and your data, and tells you to test it against real queries [7]. For it to transfer you would need the same embedding model and a query mix where the tokens that distinguish two questions move the vector far enough to fall below the threshold. The article's own counterexample is the weather in one town against the weather in another, where high similarity does not mean the answers are interchangeable [9]. Check the sign convention too, because cosine similarity rises toward 1 for closer matches while some engines report a distance that falls toward 0 [8].
A close-enough semantic hit is promoted into the exact-match store under the hash of the new query [10]. From then on that query is served by hash lookup, and the similarity comparison does not run again [2].
The article does not report a hit rate or a cost figure; the pipeline precedent is described as significantly reducing the compute the job consumes [16]. The duplication it lists is categorical: upstream users converging on similar questions, batch jobs repeating boilerplate, prompt experiments and CI runs invoking the same prompt, tool-calling agents hitting the same knowledge-base tool many times in one working day [14]. Whether tier one pays depends on how much of your traffic falls into those categories, and each team has to measure that on its own request logs.
What to watch
- A measured hit rate from a real batch pipeline would show whether normalized request bodies collide often enough to pay for the store.
- Threshold data from published deployments would test whether 0.90 to 0.95 survives a different embedding model and query mix.
- Provider-side deduplication of identical requests would remove much of the reason to run a local exact-match store.