Mercor AI Benchmark Coverage Leaves 90% of US Work Hours in the Dark

Mercor AI benchmark coverage leaves roughly 90% of American working hours with little or no meaningful representation, according to the company’s new economic map. The finding challenges a central assumption behind many claims about artificial intelligence and employment. Model scores can look impressive while saying remarkably little about most work performed across the country.

Mercor mapped 29 agentic benchmarks against 18,797 tasks spanning 923 occupations in the US labor market. Only 924 tasks met its coverage threshold. Just 208 occupations had at least one task represented by five or more benchmark prompts.

That gap does not show that AI cannot perform the remaining work. It shows that researchers have not adequately measured whether it can. The distinction matters for employers making automation decisions, workers assessing career risks, and policymakers forecasting labor disruption.

The online reaction captured the conflict quickly. Some readers treated the map as evidence that AI’s economic reach remains narrow. Others argued that missing benchmarks reveal a measurement backlog, not a capability limit.

The real contest is therefore not AI against human labor. It is benchmark confidence against the messy reality of work. Mercor’s map suggests the evidence supporting economy-wide automation forecasts remains far thinner than many confident predictions imply.

Mercor AI Benchmark Coverage Measures the Test, Not AI’s Limit

The headline number describes a measurement gap, not the percentage of work that AI cannot perform.

Mercor assembled what it calls a Benchmark Map of the US economy. The project connects prompts from 29 AI evaluations to task descriptions in the Department of Labor’s occupational taxonomy.

These evaluations emphasize agentic tasks. An agentic task asks a model to complete a deliverable or workflow, rather than answer a factual question. Examples include editing a spreadsheet, resolving a software issue, or preparing a professional analysis.

The benchmark map reports that 924 of 18,797 occupational tasks have sufficient representation. Mercor labels a task covered when an AI judge associates it with at least five prompts from the selected benchmarks.

That rule produces a stark result. Only 208 of 923 occupations contain even one task passing the five-prompt threshold. Six of the 22 occupational families contain no occupation that qualifies.

Coverage also varies sharply between fields. Computer and mathematical occupations have 29 of 36 occupations represented. Business and financial operations reach 33 of 48, while legal occupations reach five of seven.

Other parts of the economy remain much thinner. Healthcare practitioners have 12 of 89 occupations represented. Education and library work reaches nine of 62 occupations, while architecture and engineering reaches 19 of 56.

Those figures should not be read as model performance scores. An occupation can be covered even when the evaluated models perform poorly. Conversely, a task can appear uncovered even if a commercial system already assists people with it.

Coverage instead indicates whether an included benchmark offers enough prompts to support a limited evaluation. It is an inventory of available tests, not an inventory of everything current AI can do.

This distinction addresses one criticism raised in the public discussion. Restaurant automation, voice ordering, or warehouse robotics can exist without appearing in Mercor’s selected benchmark collection. Their absence does not erase deployed systems.

The same caution works in the opposite direction. A model passing prompts related to one task does not show that it can replace the corresponding worker. Jobs combine judgment, coordination, physical action, exception handling, and institutional responsibility.

A legal benchmark might evaluate contract drafting but omit client counseling, negotiation, supervision, or courtroom work. Measuring one visible deliverable does not capture the complete bundle that employers purchase when hiring a lawyer.

Mercor explicitly presents its initial list as provisional. It says the map will expand as more economically relevant evaluations become available. That limitation makes the result less definitive, but it also reinforces the central finding.

Even after years of rapid benchmark production, the evaluation ecosystem still observes a narrow slice of the economy. Claims about broad labor substitution therefore extend far beyond the available tests.

Why AI Evaluations Cluster Around Valuable Digital Work

Benchmarks concentrate where work is digital, expensive, repeatable, and relatively easy to grade.

The map’s blind spots are not randomly distributed. Mercor found that tasks classified as exposed to language models were about six times more likely to appear in its benchmark collection.

That pattern is understandable. Researchers want evaluations that can reveal differences between advanced models. Digital knowledge work offers accessible inputs, reproducible environments, and outputs that software or expert graders can inspect.

Software engineering became an early target because a program can often be tested automatically. Code either compiles, passes tests, or produces a defined output. Those feedback loops make evaluation and model improvement easier.

Financial analysis also produces familiar digital artifacts. A benchmark can provide source documents, specify assumptions, and request a spreadsheet or presentation. Reviewers can then compare calculations, structure, and conclusions.

Physical and interpersonal work creates harder measurement problems. A prompt cannot fully reproduce a crowded classroom, an unpredictable maintenance site, or a sensitive conversation with a family.

The official O*NET database helps researchers describe such work, but its task statements are not benchmark environments. They identify duties without automatically generating realistic inputs, tools, constraints, and grading rules.

Mercor’s analysis also finds that tasks from occupations with larger wage bills receive more coverage. This relationship remains after accounting for estimated AI exposure.

That incentive is economically rational. Automating part of an expensive occupation offers a clearer potential return than building an evaluation for a smaller labor market. Customers also have stronger reasons to fund tests tied to high compensation.

Task importance creates another selection effect. Within an occupation, tasks rated most important were roughly six to ten times more likely to receive benchmark coverage than the least important tasks.

Prioritizing important work sounds sensible. However, a job often depends on numerous small activities that keep its primary deliverables possible. These include gathering context, confirming requirements, coordinating approvals, detecting exceptions, and repairing misunderstandings.

A benchmark may capture the document that appears at the end of a process. It rarely captures the organizational work that determines whether the document is correct, timely, trusted, and usable.

This resembles the doorman problem raised during the Hacker News discussion. Opening a door appears to be the central task until automation reveals that the worker also provided directions, security awareness, and informal assistance.

Knowledge work contains similar hidden functions. A product manager does not merely write a requirements document. The role includes reconciling conflicting demands, remembering earlier decisions, noticing missing evidence, and securing commitment.

AI can accelerate the visible artifact while leaving those surrounding constraints intact. In some cases, faster generation creates additional review work because teams must verify a larger volume of plausible output.

That is why a personal or team knowledge base can matter more than another isolated generation tool. The bottleneck often involves recovering context and validating decisions, not producing another draft.

Benchmark concentration therefore reflects both technical feasibility and market incentives. It shows where evaluation builders can create credible tests today, not where economic transformation must stop tomorrow.

The 90% Finding Comes From Several Layers of Estimation

Mercor’s result combines official labor data with model-generated mappings, so every layer adds useful structure and additional uncertainty.

The project begins with O*NET, the federal database describing occupations, worker characteristics, and job tasks. Mercor uses 923 occupations and 18,797 associated task statements.

Researchers then connect benchmark prompts to those tasks. Mercor says an LLM judge can assign one prompt to several occupational tasks when the activities overlap.

The company validated this process against human expert labels for its own APEX benchmarks. However, that validation applies to a subset of the 29 evaluations, not necessarily every included prompt.

Mercor defines coverage at five mapped prompts. That threshold avoids treating one example as adequate representation, but five items still provide limited evidence for a complex task.

One discussion participant highlighted this problem using social work. Thirteen prompts might technically cover a counseling duty, yet they cannot represent every client, legal constraint, emotional dynamic, or adverse outcome.

The 90% estimate requires another step. O*NET records task importance and frequency, but it does not directly provide a complete allocation of every worker’s hours across every task.

Mercor therefore uses research that estimates task-level working-time shares with language models. It combines those shares with employment and wage data to approximate how American working hours are distributed.

Occupation wage bills are calculated from estimated employment and mean wages. The model assumes a 40-hour workweek when converting those totals into labor measures.

This produces a coherent economic map, but not a direct national time-use survey. The final percentage depends on task definitions, estimated time shares, benchmark selection, prompt mapping, and the five-prompt rule.

Mercor acknowledges several limitations. Its list is incomplete, and any missing benchmarks will reduce apparent coverage. The company also capped representation from heavily studied job families to prevent software engineering from dominating the collection.

That decision improves breadth while lowering measured depth in popular categories. Excluding saturated benchmarks can also hide tasks that models already perform well.

Including those benchmarks would create a different problem. Easy or unrealistic evaluations might inflate apparent economic coverage without providing useful evidence about deployment.

Selection criteria introduce judgment as well. “Economic realism” sounds objective, but researchers must still decide whether a prompt resembles work performed inside an enterprise.

The 90% number should consequently be treated as an estimate generated by an explicit methodology. It is not a census result, and it is not a fixed property of the economy.

Yet the uncertainties do not make the project meaningless. Even a substantial revision would leave the evaluation landscape uneven. Moving from 10% coverage to 15% or 20% would not establish economy-wide visibility.

The defensible conclusion is narrower and more consequential. Existing agentic benchmarks do not support confident judgments about most US working time.

That conclusion should change how readers interpret leaderboard gains. A rising score can demonstrate progress inside a tested domain while leaving thousands of other tasks unexamined.

Better Real-World Benchmarks Still Miss the Whole Job

New evaluations are becoming more realistic, but realism within selected occupations is not the same as representative coverage of the economy.

The benchmark field has already moved beyond trivia questions and academic exams. Newer tests ask models to create reports, operate software, analyze professional materials, and complete multistep assignments.

OpenAI’s GDPval evaluation illustrates that change. It covers 44 occupations across nine industries and asks models to produce work products judged against professional outputs.

The full GDPval collection contains 1,320 tasks. OpenAI selected occupations from economically significant sectors, emphasizing knowledge work tied to major wage pools.

This approach offers more economic relevance than measuring isolated reasoning questions. It also shares the concentration identified by Mercor. The first version favors valuable, digitally expressible work.

Software tests show the same pattern at a narrower level. They can measure whether an agent repairs an issue in a repository, but the result does not capture every responsibility of a working engineer.

Engineers investigate ambiguous incidents, negotiate technical debt, review changes, support colleagues, and make decisions under incomplete information. Much of that work depends on accumulated organizational context.

METR’s time-horizon research adds another useful measure. It estimates the task duration at which an agent has a given probability of succeeding.

METR also warns that these results should not be interpreted as equivalent hours of normal employment. Evaluation tasks usually contain the information needed to begin, while employees draw on earlier conversations and tacit knowledge.

That difference is central to the Mercor story. Benchmarks isolate work so researchers can grade it. Real organizations integrate work across people, systems, history, and shifting priorities.

A model can complete a well-specified assignment yet fail when requirements remain implicit. It can generate a credible spreadsheet while missing a business rule known only to an experienced analyst.

Deployment adds costs that benchmark scores often omit. Employees must prepare inputs, grant system access, review outputs, handle failures, document accountability, and decide when human escalation is required.

The model’s output can also alter downstream work. Faster code generation increases the volume reaching review and testing. Faster content generation creates more material requiring fact-checking and approval.

This is one reason measured gains do not always translate directly into organizational productivity. Automating one stage moves attention toward the slowest remaining constraint.

Physical tasks widen the gap further. Research from the remote-work estimate concluded that 37% of US jobs could be performed entirely from home. The other jobs include physical components that conventional language-model benchmarks struggle to recreate.

Robotics may eventually make those activities easier to evaluate. However, a benchmark for physical work must represent perception, manipulation, safety, environmental variation, and recovery from unexpected conditions.

A simulated success does not automatically establish safe performance in a hospital, factory, kitchen, or public road. The cost of failure can also differ greatly from a wrong answer in a digital sandbox.

The path forward is therefore broader than adding prompts. Benchmark designers need richer environments, longer workflows, human collaboration measures, and clearer links between task performance and economic value.

What the Map Changes for Employers and Workers

The map weakens confident automation forecasts while strengthening the case for narrower, workflow-specific evaluation.

An employer should not begin with a frontier model’s aggregate leaderboard position. The useful question is whether that model performs the organization’s actual tasks under realistic conditions.

That assessment starts by decomposing workflows. Teams need to identify inputs, tools, dependencies, approval requirements, common exceptions, and the consequences of an incorrect result.

A claims processor, for example, may review documents that resemble benchmark inputs. The real workflow also involves missing records, policy variations, regulatory deadlines, fraud signals, customer communication, and escalation.

Performance on one clean task says little about the system’s reliability across that entire chain. The employer must test every material transition, especially where errors become expensive or irreversible.

Mercor’s data also argues against treating occupations as indivisible units. AI adoption usually changes task composition before eliminating a complete job.

A worker might spend less time generating a first draft and more time verifying evidence. Another role might shift toward customer communication, exception handling, or coordination between automated systems.

That redistribution can still affect employment. If one person completes more work, an organization may reduce hiring or reassign staff. However, the effect depends on demand, process constraints, and the cost of supervision.

Workers should apply the same task-level thinking to career risk. A profession with heavy benchmark coverage is not automatically disappearing. It does face faster experimentation because vendors can measure and improve relevant capabilities.

Software developers, financial analysts, and lawyers sit inside this highly observed zone. Their work produces digital artifacts, attracts benchmark investment, and offers feedback that supports rapid iteration.

Less-covered occupations are not permanently protected. Their immediate uncertainty is simply greater. Benchmark scarcity prevents strong conclusions in either direction.

The map also challenges a common interpretation of low coverage. Missing tests do not prove that a task is uniquely human. They can indicate that the task is difficult to specify, grade, digitize, or fund.

Those obstacles still matter commercially. A capability that cannot be evaluated reliably is difficult to purchase, govern, insure, and deploy at scale.

For enterprise buyers, the best response is controlled evaluation using internal examples. Teams should compare model output with current work, measure review time, record failure types, and track whether gains survive across complete processes.

They should also test augmentation separately from autonomy. A model that helps an expert finish sooner can deliver value even when it cannot own the workflow.

Conversely, a model that performs well alone in a benchmark may add little inside a company. Employees might spend more time preparing prompts, checking outputs, and repairing integration failures than they save.

For policymakers, Mercor’s map highlights an evidence problem. Regulation, training investment, and workforce forecasts require more than claims about broad model intelligence.

Public institutions need transparent evaluations across representative occupations. Those tests should include physical work, public-facing services, lower-wage roles, and tasks where accountability matters more than polished output.

The map’s most useful message is therefore neither reassurance nor alarm. It is a demand for better measurement before institutions make irreversible decisions.

Three Signals Will Show Whether the Blind Spot Is Closing

The next stage of AI labor research will be judged by coverage breadth, deployment evidence, and credible tests of complete workflows.

The first signal is Mercor’s own map expansion. The company describes the current 29-benchmark index as a starting point and invites researchers to submit additional evaluations.

Watch whether new additions meaningfully expand occupational coverage. Another coding suite might deepen evidence in a familiar area without reducing uncertainty across the broader economy.

A stronger update would add credible benchmarks for education, healthcare support, administration, skilled trades, transportation, and service work. It would also report whether the 90% estimate changes under alternative coverage thresholds.

If expanded collections leave the number near its current level, the measurement problem is structural. Researchers may struggle to translate much of human work into repeatable, gradable agent tasks.

A sharp decline would weaken the headline while improving the evidence base. It would show that the initial result reflected benchmark selection more than a lasting evaluation barrier.

The second signal is independent workplace validation. Model developers and benchmark creators increasingly claim professional-level performance on selected assignments.

Those claims need comparison with deployed outcomes. Useful evidence includes total completion time, human review effort, correction rates, exception frequency, and downstream quality.

The critical unit is the complete workflow, not the generated artifact. If AI finishes drafting faster but extends review and approval, the organization has moved work rather than removed it.

Consistent productivity gains across several organizations would strengthen claims that tested capabilities transfer into real operations. Results limited to tightly controlled pilots would preserve the gap between benchmark success and economic impact.

The third signal is progress on evaluation outside pure digital knowledge work. Language models can direct tools, while robotics systems can connect planning to physical action.

Meaningful evidence would require diverse environments and transparent failure reporting. A robot succeeding in one prepared setting does not establish reliable performance across homes, hospitals, warehouses, or public spaces.

Researchers should also measure human collaboration. Most near-term systems will work alongside employees, making augmentation quality as important as autonomous completion.

A useful benchmark would test whether AI helps people make better decisions, not merely whether it produces an answer. It would include context retrieval, uncertainty communication, correction, and escalation.

These signals matter because the Mercor AI benchmark coverage gap will not close through larger model scores alone. It closes when evaluations represent more of the economy and predict actual workplace results.

For readers deciding how much confidence to place in the next automation forecast, ask three questions. Which occupational tasks were tested, how much human support was required, and did the measured gain survive real deployment?

Until those answers become routine, broad claims about AI and employment will remain ahead of the evidence. Track the map’s expansion, demand workflow-level results, and treat every leaderboard as a view through one narrow window.