The cutoff sits in the wrong place to know what it is deciding. Similarity is measured between two embeddings; correctness is a property of two answers [1]. One global number asserts that the mapping from embedding distance to answer agreement is identical everywhere in the space and for every entry in the cache. Set it low and unrelated prompts start hitting, and the retrieved response differs from the correct output [10]. Set it high and the system calls the model for answers it already holds [11]. The two failure modes pull the same scalar in opposite directions [1], so a value found by trying candidates before deployment is right on average and wrong per prompt [12]. A boundary between correct and incorrect answers is a strange thing to express as one round constant.
vCache attaches the cutoff to the entry rather than to the system: an online estimator learns a threshold for each cached prompt, and the operator supplies an error rate instead [5][4]. That changes the state you carry. A static configuration tunes one number no matter how large the cache; per-entry thresholds tune as many numbers as there are entries [2]. Storage is not the interesting cost, because the miss path in the design the paper describes already writes a threshold into the entry's metadata [9]. Exploration is. An estimator that learns online learns from observations, and a fresh entry has none, so early traffic near it pays model calls to buy confidence that later hits spend.
The two headline multiples in the abstract are maxima, measured against static-threshold and fine-tuned embedding baselines on the four benchmarks the authors release alongside the code [6][7]. For the hit-rate side to transfer, your traffic needs the paraphrase density those benchmarks carry: the same intent arriving in different words, which is precisely where exact string matching fails [14]. For the error side to transfer, your definition of a wrong answer has to match whatever the experiments scored against, and the excerpt available does not say how that label is produced at serving time [16]. The paper also scopes semantic caching to single-turn requests with short to medium context, such as search queries and classification tasks [13], so a long-context agent loop is outside the claim being made.
One asymmetry is what makes a bound worth paying for. At the 100x latency reduction the paper cites for semantic caching [8], a false positive reaches the user around two orders of magnitude sooner than the correct answer would have [3]. In my context I would take the per-entry estimator with a conservative bound, and log the learned threshold and the observed similarity on every hit, because the failure I care about is the one that never produces a metric.
