Build1 publisher2 min readPublished
Handed the right tool in a five-item shortlist, a small model picks it 19 times in 20
A two-stage eval forces every shortlist to hold the correct tool plus its four strongest BM25 siblings. Selection accuracy comes in at 90 to 97 percent. That puts the 5 percent paraphrase recall on the critical path.
The Engineer · Build desk
What happened
- Stage two of the eval scored tool selection with a live model on forced shortlists, because scoring the retriever's own output would have measured retrieval a second time.
- gpt-5.4-mini ran at its default temperature over sixty tasks per condition, half phrased in the tool's own vocabulary and half paraphrased, across the four description conditions carried over from the recall test.
- Selection accuracy landed between 90 and 97 percent in every condition tested, on a shortlist of five.
- Verbose descriptions, which had done nothing for recall, did nothing for selection either, and terse ones held their own against description sets costing three times as many tokens.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision The reflex when an agent picks the wrong tool is to buy a bigger model, and the author argues that at a five-tool shortlist the spend goes to the stage that was holding at 90 percent.
- cost On the author's account the repair is a thirteen-token alias sentence per tool description, against a model that costs ten times more per call.
- precedent Any tool-calling benchmark that publishes one blended number keeps charging the model for the retriever's miss, and the model takes the blame until recall is reported on its own.
- constraint The result is bounded by a five-item lexical shortlist and a lenient argument check, so teams validating arguments against a strict schema are outside what was measured.
BM25 picks the distractors. That makes them the expected tool's same-noun siblings: when the answer is approve_claim, the shortlist also holds cancel_claim, update_claim and search_claim [3]. Topic matching buys nothing in that list. The model has to read the verb, and it still picks correctly about nineteen times in twenty [10].
End to end, the post models task success as recall times selection [14]. Stage one measured 5 percent recall on paraphrased requests [11]. Multiply that through the selection band: 0.05 x 0.90 is 4.5 percent, and 0.05 x 0.97 is 4.85 percent [1]. Ninety-five of every hundred paraphrased requests never put the right tool in front of the model at all, and the selection stage takes at most half a point more [2].
Several things have to hold before that band applies to your agent. Your shortlist has to be five items, because that is the size that was scored [2]. It also has to come off a lexical retriever: the distractor set is defined by BM25 scores [2]. And the argument checking has to be as forgiving as this one: the task's reference value, an invoice number like INV-2005, only had to land somewhere in the arguments of a correct pick, and the author calls the check deliberately lenient because exact schema validation would punish harmless formatting [5]. Under strict validation, the reported 87 to 97 percent for argument filling is a ceiling [7].
The cells are thin. Sixty tasks per condition split half vocabulary and half paraphrase leaves thirty per cell, where one or two tasks move any difference in the table, and the author notes that one run at default temperature is a sample, not a truth [8]. "Read that table as flat, and I mean that as a finding, not a shrug," the author wrote [9].
The paraphrase result is the useful half. The gap that wrecked retrieval nearly disappears at selection, and with approve_claim on the list the model gets there from "sign off on the damage report" without difficulty [13].
The eval refused to run with an API key, because the Azure AI resource had disableLocalAuth set, so the harness built to measure keyless MCP tooling had to authenticate with an Entra bearer token itself [18].
The runner is measure/eval_selection.py in the companion repo, stdlib Python against any Azure OpenAI deployment, with every result row tagged by the description-set hash so numbers stay attributable across description changes [19].
What to watch
- The author names a shortlist of twenty near-duplicates or genuinely ambiguous intents as the next measurable test of judgment; that run would show where selection starts to break.
- A dense or embedding retriever on the same pools would test whether the 5 percent paraphrase recall is a property of BM25 or of the descriptions. That recall sets the end-to-end number.
- The same forced shortlists against a larger model would show whether 90 to 97 percent is the model's ceiling or the harness's.