A benchmark map exposes gaps between AI tests and workplace tasks

Mercor maps prompts from 29 selected AI benchmarks onto US occupational tasks, using model-generated tags checked against human tags on a subset. Requiring at least five prompts per task, its index covers 924 of 18,797 tasks across 208 of 923 occupations. This measures coverage of a chosen benchmark collection, not what AI can actually do; the authors warn that their incomplete selection can understate coverage.

Mercor Research

Why it matters

A benchmark score is easier to interpret when you know which parts of the intended workflow the benchmark never tests.

More context

The public map exposes benchmark selection criteria and its occupational taxonomy. Selection favors unsaturated, agentic tasks and caps benchmarks per job family. Use the methodology as a prompt to inventory missing tasks in your own evaluations. The exact original publication date was not established.

Keep in mind

Task matching and working-time estimates partly rely on language models. The provisional index is not representative of every benchmark, and coverage does not establish reliable task completion.

Original read