Q2D-Web scores your retriever against labels only Perplexity can see

Submission works like this. You point an evaluation form at a publicly available Hugging Face checkpoint, Perplexity runs it, and a score lands on a public leaderboard ^[9]. The corpus, the queries and the relevance labels stay with Perplexity ^[8]. That gives you a regression signal and not much else, because improving a first-stage retriever is mostly reading the queries you lost and deciding whether the miss came from chunking, a truncated max sequence length, or a tokenizer that treats "2.14.1" as noise. Perplexity acknowledges the cost directly: outside researchers cannot reproduce a full run, and cannot inspect individual examples where the scoring may have made a questionable relevance call ^[10].

Memorization is the stated reason for keeping the set closed. Release the test set and it eventually reaches training corpora, at which point the leaderboard rewards recall of the answer key rather than retrieval ^[11]. That failure mode is real, and the paper is honest that the fix makes Perplexity the permanent evaluator ^[10].

Worth studying is how the corpus itself was built. It is the top 5,000 production retrieval results for each query, deduplicated with MinHash-LSH down to roughly 190 million pages ^[12]. Multiply 69,721 queries by 5,000 and you get about 349 million document slots gathered, so around 45 percent collapsed under dedup and cross-query overlap ^[19]. What survives is dense in near-misses: pages on the right topic that omit a required date, product version or named entity ^[13]. Labeling runs deep in the same direction, averaging 99.6 positive judgments per query in the combined set ^[16], which works out to roughly 6.9 million positives across the query set ^[20]. That mostly removes the failure where your model surfaces a genuinely good page the benchmark never labeled and eats a penalty for it.

Good engineering shows up in the pool hygiene. Models released before January 1st, 2025 built the evaluation pool and generated labels, while every evaluated neural checkpoint postdates that cutoff, with BM25 declared as the exception because it did both jobs and served as the lexical baseline ^[17]. That is the discipline that stops a benchmark from quietly rewarding the model family that chose its own positives.

For the score to transfer, several things have to hold on your side. Your queries have to be machine-written the way an agent writes them, after rewriting the request, folding in earlier turns and decomposing the task into narrower searches ^[5]; roughly three of them per user search here ^[22], and 82 percent of the set is supporting queries chasing background and adjacent entities rather than the user's actual question ^[21]. Your language mix has to tolerate 65.8 percent English ^[7]. And your definition of relevance has to resemble theirs, which unions pages a production agent cited with up to 50 results per query from Perplexity's internal web-ranking stack, plus DeepSeek-V4-Flash judgments on unlabeled documents ^[14]. Two of those three signals are Perplexity's own retrieval and reranking choices, a bias the paper names ^[15]. If you retrieve over an enterprise corpus with one correct document per query, a test averaging about a hundred positives is measuring a different job.

Several of the authors also built pplx-embed ^[4], so the group working on the first retrieval stage holds the only copy of the ruler. Runtimewire heads its results section "Perplexity's model wins Perplexity's test, mostly", and the per-model figures behind that line are not in the account ^[18]. The number that matters is the margin, and the leaderboard is the only place it can be read.