Skip to content

Build Also reported elsewhere2 publishers 2 min read Published

Mercor's map of 29 AI benchmarks covers just 208 of 923 US occupations

Mercor's Benchmark Map finds only 208 of 923 US occupations have a task with five or more AI benchmark prompts mapped to it. The report says about 90% of paid hours fall in tasks benchmarks barely cover, so leaderboard scores speak to a thin slice of work.

The Engineer · Build desk

Send a correction

Illustration accompanying Mercor's map of 29 AI benchmarks covers just 208 of 923 US occupations
Generated illustration
Benchmark coverage tracks salary, the report says Occupations with a task carrying five or more mapped benchmark prompts, and occupational families with no occupation really covered.

Share chart: 208 of 923 US occupations have a task with five or more mapped benchmark prompts. In the Computing and Mathematics family, 29 of 36 measured occupations are reached. 6 of 22 occupational families, including health and education, have no occupation with real coverage.

Benchmark coverage tracks salary, the report says
SharePartWholePercentClaim
Computing and Mathematics29 occupations36 occupations81% (computed)5
All US occupations208 occupations923 occupations23% (computed)1
Families with no real coverage6 families22 families27% (computed)4

What happened

  • Six of the 22 occupational families, including health and education, have no occupation with real benchmark coverage.
  • Coverage tracks salary, according to the report: the Computing and Mathematics family reaches 29 of 36 measured occupations.
  • Mercor capped the number of benchmarks per family so that software engineering and academic research would not dominate the 29-benchmark index.
  • The index is public and interactive, with each occupation shown as a cell colored by the share of its tasks that have coverage.

Why it matters

  • Who has to decide decision A team deploying AI in a role among the 715 uncovered occupations has little public benchmark evidence to lean on and has to budget for its own evaluation.
  • What it limits constraint The 208 counts occupations, not tasks, so it cannot say how much of any single job the benchmarks actually test.
  • What it sets up precedent Mercor plans continuous updates, so the 208 and the 90% are dated figures and should be cited with the map version.

The measurement rests on a language-model judge. According to DEV Community's write-up of the report, the judge reads each prompt from the 29 benchmarks and assigns it one or more of the 18,797 O*NET tasks [3]. Any error in that mapping moves the count directly. Of the 923 occupations, 715 have no task that clears the five-prompt bar [13]. The 208 that do are 22.5% of the total [14][1].

The hours figure travels less well. Mercor's report says nine of ten paid work hours fall in tasks with no benchmark or only minimal coverage [2]. For that to hold on your workload, your tasks would have to resemble the national mix of paid hours, and the judge's mapping would have to survive the wording of your task descriptions.

We'd expect software teams to be the exception. The report itself says benchmark saturation in software engineering is no accident: tests, clear success metrics and public repositories make evaluation easy to automate, and AI labs want to measure their own models' progress [7].

Mercor's position is worth stating. It runs a marketplace that connects experts from many professions with AI labs to produce training and evaluation data [8]. Its own benchmark, APEX, is one of the 29, and the company has unpublished prompts it can map directly to each task [9]. In our view, a map of where evaluation is thin is also a map of where Mercor's product is needed. The index is public, so the mapping can be checked [12].

Mercor also crosses the binary exposure labels from a 2023 study by Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock with its coverage map, to test whether benchmarks cluster where exposure is highest [10]. The write-up does not report that result.

What to watch

  • Whether Mercor's updates add benchmarks for the six families with no covered occupation, and how far the 208 moves.
  • Whether Mercor publishes its test of benchmark coverage against the 2023 exposure labels.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence45
Adoption
Not scored
Hype gap+15
Incentives60
Confidence50
Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

  1. [1]

    Mercor reports that only 208 of the 923 US occupations have at least one task with five or more prompts from some benchmark mapped to it.

    Reported Supported Source: Mercor Benchmark Map report, as summarized by DEV CommunityView cited source
  2. [2]

    Mercor's report says nine of ten paid work hours in the US fall in tasks with no benchmark or minimal benchmark coverage.

    Reported Supported Source: Mercor Benchmark Map report, as summarized by DEV CommunityView cited source
  3. [3]

    Mercor uses a language model as an automatic judge that reads each prompt of the 29 benchmarks and assigns it one or more of the 18,797 O*NET tasks.

    Reported Supported Source: DEV Community's description of Mercor's methodView cited source
  4. [4]

    Six of the 22 occupational families, including health and education, have no occupation with real coverage.

    Reported Supported Source: Mercor Benchmark Map report, as summarized by DEV CommunityView cited source
  5. [5]

    Coverage tracks salary: the Computing and Mathematics family reaches 29 of 36 measured occupations.

    Reported Supported Source: Mercor Benchmark Map report, as summarized by DEV CommunityView cited source
  6. [6]

    Mercor imposed a cap on benchmarks per occupational family when building its 29-benchmark index, so that software engineering and academic research would not dominate as they do in earlier catalogs.

    Reported Supported Source: DEV Community's summary of Mercor's reportView cited source
  7. [7]

    The report acknowledges that benchmark saturation in software engineering is no accident: it is the task type where evaluation is easiest to automate, with tests, clear success metrics and public repositories, and where AI labs have a direct interest in measuring their models' progress.

    Reported Supported Source: Mercor Benchmark Map report, as summarized by DEV CommunityView cited source
  8. [8]

    Mercor operates a marketplace that connects experts from different professions with AI labs to generate training and evaluation data.

    Reported Supported Source: DEV CommunityView cited source
  9. [9]

    Mercor's own benchmark, APEX, is one of the 29 benchmarks in the index, and the company has access to unpublished prompts that it can map directly to each O*NET task.

    Reported Supported Source: DEV CommunityView cited source
  10. [10]

    Mercor crosses the binary exposure labels from a 2023 study by Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock with its own coverage map to see whether AI benchmarks concentrate where exposure is highest.

    Reported Supported Source: DEV Community's summary of Mercor's reportView cited source
  11. [11]

    Mercor says the Benchmark Map is only a first step toward a resource it plans to update continuously.

    Reported Supported Source: Mercor, as summarized by DEV CommunityView cited source
  12. [12]

    The Benchmark Map is a public index available as an interactive map, in which each occupation appears as a cell colored by the percentage of its tasks with coverage.

    Reported Supported Source: DEV CommunityView cited source
  13. [13]

    715 of the 923 occupations have no task with five or more mapped benchmark prompts.

    Derived Supported
  14. [14]

    The 208 covered occupations are about 22.5% of the 923.

    Derived Supported

Sources

2 independent publishers whose own reporting we read for this story.

  1. dev.to

    1 article · October 16, 2026

    Mercor: solo 208 de 923 ocupaciones con cobertura real de IA
  2. runtimewire.com

    1 article · October 14, 2026

    Mercor maps AI benchmarks to jobs and finds most US work unmeasured

Topics and entities

Entities