build1 distinct publisher
Benchmark inflation, measured: up to 16 points, and no way to tell which model is padded
A paper rebuilt a hidden holdout for TruthfulQA and found some of twenty models score as much as 16 points higher on the public version. A flat discount will not repair your shortlist.
Publishers:arxiv.org
Reality
- Evidence58
- Adoption20
- Hype gap+22
- Incentives48