4.5 Million Graded Answers Say the Fact Is on the Shelf and the Model Cannot Find It

The figure is a sample size, not a score: roughly 4.5 million graded model responses, produced by Google Research scientists Nitay Calderon and Gal Yona to separate facts a model never learned from facts it learned and cannot retrieve ^[3][12]. That split matters because the two failures have different fixes and different bills.

The arithmetic is worth seeing, because it bounds what the number supports. WikiProfile holds 2,150 facts extracted from Wikipedia pages, each paired with ten tasks: two probing encoding, four probing knowledge, four multiple-choice items probing recognition ^[4][14]. Thirteen models were evaluated, each with and without thinking, with eight responses sampled per model, fact and task [c3b][c3c]. That product is about 4.47 million responses ^[7]. The scale comes from repetition and task variants, not breadth of subject matter: 2,150 facts examined very hard, with the arXiv abstract citing 4 million responses across 13 models ^[2].

What it turns on: for Gemini-3-Pro and GPT-5, 95 to 98 percent of facts are encoded, and yet those same models fail to directly recall 26 to 34 percent of them ^[5]. With thinking, they still fail on 11 to 12 percent ^[6]. Comparing the reported range endpoints, thinking closes roughly three fifths of the direct-recall gap ^[8]. The authors report the failures are systematic, falling disproportionately on long-tail facts and reverse questions ^[9].

The decision this changes is which remedy to fund. Encoding failures call for scaling model size or expanding data coverage; recall failures point instead to post-training and inference-time methods that use what is already stored ^[10]. The paper's conclusion is that future factuality gains may depend less on scaling than on utilisation ^[13].

One caution on the instrument: the benchmark was built by an automated pipeline driven by a prompted Gemini-2.5-Pro with thinking, filtered against a search engine, then manually validated, and responses were graded by prompted LLM autoraters ^[11][c3c].