GPT-4.1 falls from 93.49% to 14.58% on BrowseComp-Plus when BM25 does the searching

The cleanest way to read the table is to compare gaps. Swapping GPT-4.1 for Qwen3-32B moves the score by 10.24 points with documents supplied and by 11.09 points under BM25 ^[14][15]. Swapping supplied documents for BM25 search moves it by 78.91 points for GPT-4.1 and 79.76 for Qwen3-32B ^[12][13]. On this benchmark the retrieval condition is worth roughly seven times the model choice ^[17].

In relative terms the penalty is uneven. GPT-4.1 keeps about 16% of its supplied-document score under BM25, and Qwen3-32B keeps about 4% ^[16].

That is a measurement of two models and one baseline retriever on a fixed corpus. For it to transfer, a production corpus would need the traits the article lists: duplicated documents, outdated drafts that stay searchable, different terminology for the same thing, and several versions of one policy ^[18]. The production retriever would have to be no better than BM25. Your questions would also have to look like the benchmark's 830 research questions ^[3]. The production conclusion is the article's inference, and the article does not include production failure data.

The article's advice follows from the design: "The lesson is practical: before changing the model, examine what reaches it." ^[10] We think that is the right first step, with one condition. It needs a log of what the retriever returned for each failed task, or the check cannot be run.

Retrieval quality also interacts with latency. If an agent runs ten searches in parallel and needs all ten before it continues, the slowest one sets the pace for that step, and repeating the pattern makes tail latency matter more than it does for a single interactive query ^[8]. Because better retrieval also reduced search calls ^[6], we'd expect it to cut the number of times a task waits on that slowest search.