Semantic prompt caching

Learning one similarity threshold per cached prompt buys a bound on wrong hits

A single cosine cutoff is one number standing in for a question that is different at every cache entry, and the vCache authors report both higher hit rates and lower error rates once each entry learns its own.