Build1 publisher2 min readPublished Updated
Paraphrased requests drop BM25 tool recall to 5 percent across a 100-tool pool
Deferred tool loading assumes retrieval puts the right tool in the shortlist. A BM25 harness over 100 synthetic enterprise tools and 200 tasks says it does that 5 percent of the time when the user does not use your words.
The Engineer · Build desk
What happened
- A dev.to post split agent tool selection into a recall stage and a selection stage after a reader comment framed it that way, then tested recall only, because you can measure recall without a live model.
- The harness scores BM25 over the serialized definitions of 100 synthetic enterprise tools at terse, realistic and verbose description lengths, with a shortlist of five and 200 tasks, two per tool.
- On requests that echo a tool's own domain words, retrieval is effectively solved, and even terse descriptions score perfectly because the tool name alone carries the match.
- On paraphrased requests that share the intent and the identifier but no content word, recall at five falls to 5 percent, and the verbose tier at 208 tokens per tool scores statistically the same.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A retrieval miss raises no error. The agent answers from the wrong five tools, and the trace looks like a cheap, successful turn, so nothing in monitoring fires.
- cost Buying recall with shortlist width is a recurring bill: at this pool size it runs near 100 tokens per recall point, charged on every call of every session.
- decision Whoever writes the tool descriptions now has a cheaper place to spend the budget: the words users actually type, instead of another paragraph drawn from the same domain vocabulary.
BM25 is a lexical scorer. It ranks each tool by the words it shares with the request, and the documents it scores here are the serialized tool definitions: name, description, parameter descriptions [7]. So the terse tier wins the easy case. When the request says "Please approve the claim CLM-2041." [8], approve_claim is a direct term match, and the post reports that the tool name alone carries it [9].
The paraphrase tasks were built to remove that overlap. "Can you sign off on that damage report, number CLM-2041?" shares the intent and the identifier with the approval tool and not one content word [8]. Recall at a shortlist of five is 5 percent [10]. The post describes the gap in plain vocabulary terms: users say "bill", "delivery", "sign off" while the descriptions say "invoice", "shipment", "approve" [12].
Then the width sweep. Five realistic tools cost about 542 tokens per model call and ten cost 1,040 [13], and the extra five slots move recall from 5 to 10 percent [14]. Call it 498 additional tokens in every call for five points, close to 100 tokens per point [1]. The slope is one point per tool added, so closing the gap by width alone implies a hundred-tool shortlist [2] at roughly 10,400 tokens per call [3]. That is the whole manifest deferred loading exists to avoid shipping [20].
The alias line prices differently. At 13.4 tokens per tool [17], a five-tool shortlist carries about 67 extra tokens per call [4], and the alias condition already reaches 100 percent recall at a shortlist of three [15]. The verbose tier's 97 extra tokens per tool [17] is 485 per call at the same shortlist [5], for recall statistically identical to realistic [11]. The post's explanation is that the extra words repeat the same words [21].
Both endpoints of this eval are constructed, and that governs what transfers. The 5 percent floor comes from tasks written to share no content word with the description [8], and the author states the caveat directly: the alias sentences contain the same synonyms the paraphrase tasks use, so 100 percent is the ceiling case [18]. Real traffic sits somewhere in between. The method is what transfers: stage one runs without a model [3], so a team can run its own retriever over its own manifest against logged user phrasings and get a recall number before paying for a single selection run [4]. The harness here is BM25 [7], and the post does not test an embedding retriever.
What to watch
- Stage two, selection from the shortlist, still needs live model runs and has not been published.
- An alias set built from logged production phrasings rather than the eval's own synonyms would give a non-ceiling recall number.
- Whether an embedding retriever narrows the paraphrase gap without alias lines. This BM25 harness does not answer that.