GPT-4.1 falls from 93.49% to 14.58% on BrowseComp-Plus when BM25 does the searching
GPT-4.1 scored 93.49% on BrowseComp-Plus with documents supplied and 14.58% when BM25 searched, according to The New Stack. That 78.9-point gap sizes the room for retrieval work on one fixed corpus. Production failure rates need their own measurement.
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap+15
- Incentives
- Insufficient
- Confidence55