Topic
Caching LLM responses by meaning similarity over embeddings rather than exact input hashes.
A dev.to writeup argues a predictive router with a semantic cache beats a cheap-model-first cascade on interactive traffic, and the flow it publishes keeps verifier-backed double generation for every medium-confidence query.
Reality
A response-caching walkthrough in The New Stack puts model settings and upstream data inside the cache key, so a version bump flushes the store by design. The semantic tier layered on top is where wrong answers enter.
Reality
The provider cache discounts a repeated prefix, and the worst case is bounded arithmetic. The semantic cache deletes the call outright, but a miss there means a wrong answer, not a rounding error, so the threshold sweep matters more than the hit rate.
Reality
Six days after a reviewer asked him to prove a 0.92 similarity threshold was safe, the author switched his cache off. Then he found two of his own test labels were wrong.
Reality
A dev.to tutorial builds a semantic cache in pure Python. The part worth your time is not the code, it is what hit rate does to your effective quota under a rate limit.
Reality
A runnable Mem0-backed wrapper blocks an agent's repeat tool call before it burns the request. The pattern is sound; the default policy will block you forever.
Reality
A dev.to writeup traces a support bot's dead cache to hash-based lookup and replaces it with cosine similarity over embeddings at a 0.92 threshold. The lever sits in retrieval, not model choice.
Reality