Mercor's map of 29 AI benchmarks covers just 208 of 923 US occupations
Mercor's Benchmark Map finds only 208 of 923 US occupations have a task with five or more AI benchmark prompts mapped to it. The report says about 90% of paid hours fall in tasks benchmarks barely cover, so leaderboard scores speak to a thin slice of work.
The Engineer · Build desk
Share chart: 208 of 923 US occupations have a task with five or more mapped benchmark prompts. In the Computing and Mathematics family, 29 of 36 measured occupations are reached. 6 of 22 occupational families, including health and education, have no occupation with real coverage.
- Computing and Mathematics, claim 5 29 of 36 occupations 81% computed
Computing and Mathematics The rest of the 36
- All US occupations, claim 1 208 of 923 occupations 23% computed
All US occupations The rest of the 923
- Families with no real coverage, claim 4 6 of 22 families 27% computed
Families with no real coverage The rest of the 22
| Share | Part | Whole | Percent | Claim |
|---|---|---|---|---|
| Computing and Mathematics | 29 occupations | 36 occupations | 81% (computed) | 5 |
| All US occupations | 208 occupations | 923 occupations | 23% (computed) | 1 |
| Families with no real coverage | 6 families | 22 families | 27% (computed) | 4 |
What happened
- Six of the 22 occupational families, including health and education, have no occupation with real benchmark coverage.
- Coverage tracks salary, according to the report: the Computing and Mathematics family reaches 29 of 36 measured occupations.
- Mercor capped the number of benchmarks per family so that software engineering and academic research would not dominate the 29-benchmark index.
- The index is public and interactive, with each occupation shown as a cell colored by the share of its tasks that have coverage.
Why it matters
- Who has to decide decision A team deploying AI in a role among the 715 uncovered occupations has little public benchmark evidence to lean on and has to budget for its own evaluation.
- What it limits constraint The 208 counts occupations, not tasks, so it cannot say how much of any single job the benchmarks actually test.
- What it sets up precedent Mercor plans continuous updates, so the 208 and the 90% are dated figures and should be cited with the map version.
The measurement rests on a language-model judge. According to DEV Community's write-up of the report, the judge reads each prompt from the 29 benchmarks and assigns it one or more of the 18,797 O*NET tasks [3]. Any error in that mapping moves the count directly. Of the 923 occupations, 715 have no task that clears the five-prompt bar [13]. The 208 that do are 22.5% of the total [14][1].
The hours figure travels less well. Mercor's report says nine of ten paid work hours fall in tasks with no benchmark or only minimal coverage [2]. For that to hold on your workload, your tasks would have to resemble the national mix of paid hours, and the judge's mapping would have to survive the wording of your task descriptions.
We'd expect software teams to be the exception. The report itself says benchmark saturation in software engineering is no accident: tests, clear success metrics and public repositories make evaluation easy to automate, and AI labs want to measure their own models' progress [7].
Mercor's position is worth stating. It runs a marketplace that connects experts from many professions with AI labs to produce training and evaluation data [8]. Its own benchmark, APEX, is one of the 29, and the company has unpublished prompts it can map directly to each task [9]. In our view, a map of where evaluation is thin is also a map of where Mercor's product is needed. The index is public, so the mapping can be checked [12].
Mercor also crosses the binary exposure labels from a 2023 study by Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock with its coverage map, to test whether benchmarks cluster where exposure is highest [10]. The write-up does not report that result.
What to watch
- Whether Mercor's updates add benchmarks for the six families with no covered occupation, and how far the 208 moves.
- Whether Mercor publishes its test of benchmark coverage against the 2023 exposure labels.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+15
- Incentives60
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Mercor reports that only 208 of the 923 US occupations have at least one task with five or more prompts from some benchmark mapped to it.
- [2]
Mercor's report says nine of ten paid work hours in the US fall in tasks with no benchmark or minimal benchmark coverage.
- [3]
Mercor uses a language model as an automatic judge that reads each prompt of the 29 benchmarks and assigns it one or more of the 18,797 O*NET tasks.
- [4]
Six of the 22 occupational families, including health and education, have no occupation with real coverage.
- [5]
Coverage tracks salary: the Computing and Mathematics family reaches 29 of 36 measured occupations.
- [6]
Mercor imposed a cap on benchmarks per occupational family when building its 29-benchmark index, so that software engineering and academic research would not dominate as they do in earlier catalogs.
- [7]
The report acknowledges that benchmark saturation in software engineering is no accident: it is the task type where evaluation is easiest to automate, with tests, clear success metrics and public repositories, and where AI labs have a direct interest in measuring their models' progress.
- [8]
Mercor operates a marketplace that connects experts from different professions with AI labs to generate training and evaluation data.
- [9]
Mercor's own benchmark, APEX, is one of the 29 benchmarks in the index, and the company has access to unpublished prompts that it can map directly to each O*NET task.
- [10]
Mercor crosses the binary exposure labels from a 2023 study by Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock with its own coverage map to see whether AI benchmarks concentrate where exposure is highest.
- [11]
Mercor says the Benchmark Map is only a first step toward a resource it plans to update continuously.
- [12]
The Benchmark Map is a public index available as an interactive map, in which each occupation appears as a cell colored by the percentage of its tasks with coverage.
- [13]
715 of the 923 occupations have no task with five or more mapped benchmark prompts.
- [14]
The 208 covered occupations are about 22.5% of the 923.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
2 independent publishers whose own reporting we read for this story.
- dev.toMercor: solo 208 de 923 ocupaciones con cobertura real de IA
1 article · October 16, 2026
- runtimewire.comMercor maps AI benchmarks to jobs and finds most US work unmeasured
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI BenchmarksFollow
- Simulation Evaluation and Accuracy ClaimsFollow
- AI labor-market exposureFollow