top of page

METR Update Reveals a Gap Between AI Coding and Judgment

2 hours ago
12 min read
Illustrative photograph. Photo: Pixabay / Pexels.

METR expanded its AI capability evaluations on November 20, 2026, finding continued exponential progress alongside weaker performance on tasks requiring judgment. The METR time horizon update puts the doubling time for measured task horizons at roughly 3.6 months. But that trend does not establish how long an AI can work safely without supervision, or whether it can independently direct useful research.

The distinction matters because completing a defined programming assignment and choosing a worthwhile experiment require different evidence of success. A program can pass tests while its broader purpose remains poorly chosen. METR’s expanded evaluation suite examines both sides of that divide, with stronger results where agents receive clear feedback.

The findings put pressure on claims that coding progress translates directly into autonomous AI research. They also challenge the opposite assumption, that weaknesses in judgment guarantee a lengthy pause in research automation. METR sees substantial capabilities, meaningful limitations, and growing difficulty measuring the relationship between them.

What the METR Time Horizon Update Actually Measures

The chart tracks task difficulty through human completion time, rather than measuring an agent’s dependable working hours.

A time horizon is the human task duration associated with a specified model success rate across an evaluation suite. At the commonly discussed 50% threshold, the model is predicted to complete tasks of that difficulty about half the time. The hours describe the human reference workload, rather than the time the model spends running.

METR’s task completion horizons therefore require a more careful reading than a headline about agents working through an entire shift. A longer horizon indicates better performance on the measured distribution of tasks. It does not certify reliable performance on every assignment that takes a person that long.

The November report extends the organization’s earlier software-focused evaluations with tasks targeting capabilities relevant to AI research and development. Its results cover public models through Anthropic’s Opus 5.5 and OpenAI’s GPT-6 Astra. These models operate as agents, meaning they use tools and take successive actions toward an assigned objective.

The report covers three distinct suites. Time Horizon 1.1 contains 222 tasks, MirrorCode contains 75 software replication tasks, and TH2 contains 36 tasks across 12 families. Those counts matter because they describe different collections, rather than interchangeable versions of the same benchmark.

TH2 includes eight families with verifiable outcomes and four involving harder-to-verify research or judgment. Verifiable tasks have outcomes that evaluators can check against a defined objective. Judgment tasks require useful choices when clear, fast feedback is unavailable.

Across the combined Time Horizon 1.1 and TH2 suites, METR estimates that 50%-success time horizons double roughly every 3.6 months. That is an exponential trend in this particular measure. It does not mean every AI capability improves at the same rate.

Adding harder tasks also changes the chart’s interpretation. METR reports lower horizons on the combined suite than on Time Horizon 1.1 alone for a given model. A lower measurement after expanding the test does not mean the model lost capability.

The expansion responds to a practical problem: earlier tasks became less informative as models improved. METR says the older suite had become saturated, limiting its ability to distinguish stronger systems. The updated tests seek more difficult work and a broader capability profile.

That makes this update more than another point on a familiar rising line. It changes the questions evaluators can ask about that line. The central issue becomes whether gains on defined assignments extend to work where identifying success is part of the assignment itself.

Autonomous Research Claims Face a Different Test

The immediate pressure falls on claims about research autonomy, where producing an artifact is only part of delivering useful work.

AI research includes implementation, experimentation, interpretation, and decisions about what deserves further effort. Progress in one activity can accelerate a team without replacing the others. The November findings make that distinction harder to ignore.

METR reports that current frontier models often match or outperform the skilled human participants it recruited for coding and optimization tasks. Optimization means improving a specified score or objective under given constraints. These settings often give agents a direct way to compare attempts and keep improving.

That strength has practical significance even when the agent does not choose the broader research agenda. Researchers can delegate implementation while retaining responsibility for experimental priorities. More capable implementation can increase the amount of work a team completes.

The comparison with human experts remains narrower than a universal claim of superiority. METR acknowledges uncertainty about whether its recruited participants match frontier laboratory employees within their own specialties. Recruiting people with the necessary expertise and availability becomes difficult as assignments grow longer.

Where larger historical comparison groups exist, the results also preserve a place for exceptional human performance. METR finds strong model placements in programming optimization contests, while the best human participants generally remain ahead. In training-script optimization, top contributors likewise achieved improvements beyond the strongest agent runs.

OpenAI’s November 18 account of research acceleration offers a related, company-reported view from inside a laboratory. It reports rising success on delegated research tasks, with substantial variation between activities. Humans remain central to its account of the development process.

That evidence supports a picture of uneven automation across research workflows. It does not supply an interchangeable validation of METR’s results. Internal workplace tasks, benchmark assignments, and retrospective success assessments use different methods and capture different conditions.

For Anthropic, OpenAI, and other developers, the harder question concerns what happens after implementation becomes easier. Can agents identify which experiment would meaningfully reduce uncertainty? Can they recognize when a positive result reflects a weak test?

Enterprise buyers face a related distinction when evaluating agents for technical work. A successful demonstration with a clear acceptance test supports that specific use case. It provides less evidence about assignments involving ambiguous goals, changing requirements, or delayed consequences.

The METR time horizon update raises the standard for these claims without dismissing the underlying gains. A persuasive account of autonomous research needs evidence about direction and evaluation alongside execution. Coding performance is a meaningful component of that account, but the report does not establish that it completes the picture.

Strong Coding Does Not Settle the Judgment Question

METR’s most revealing failures appeared when models had to decide what made an evaluation useful.

One TH2 task asked agents to create an evaluation that would inform METR’s assessment of model capabilities. This required more than writing functioning software. The proposed test also needed to answer a question that METR considered valuable.

According to the capability evaluation findings, none of the agent-created tasks warranted inclusion in the report as useful new evaluations. Some outputs approached usefulness, indicating progress over weaker models. But improvement within the scoring system did not translate into work METR chose to adopt.

The agents generally favored automatically scored optimization tasks. They continued doing so in follow-up experiments that explicitly requested tasks involving decisions with poor short-term feedback. That behavior exposed a mismatch between producing a measurable challenge and identifying an informative research question.

METR also reports that agents did not independently investigate what kinds of tasks the organization would find most valuable, despite internet access. The researchers did not observe novel threat modeling or especially insightful hypotheses behind the proposed evaluations. These are observations from the tested setup, rather than evidence of a permanent inability.

The distinction is concrete. An evaluation can run correctly and still tell researchers little they need to know. Its implementation quality and its research value are separate properties, even though both influence the final result.

A second experiment examined whether models could judge the evaluations that other models produced. Strong judges agreed more closely with human ratings when assessing weaker creators. When creators were similarly strong or stronger, the judges tended to assign overly generous scores.

This finding complicates a common proposed solution to weak oversight: asking another AI to review the work. A reviewer can identify obvious defects while missing more convincing mistakes. Agreement between an author and reviewer therefore does not independently establish quality.

METR offers more than one explanation for the pattern. Models might lack the judgment needed to assess these tasks, or stronger creators might produce outputs that weaker judges find misleading. The experiments do not conclusively separate those possibilities.

The broader mechanism concerns feedback. Reinforcement learning, training that rewards outcomes under a scoring system, benefits from useful signals about which attempts worked. Coding and optimization often provide such signals through tests or measured performance.

Open-ended research can delay that feedback or make it expensive to obtain. A promising proposal might fail only after substantial experimentation. A seemingly unproductive direction might prove useful when a researcher identifies an overlooked assumption.

The report does not establish that judgment and coding are fundamentally separate abilities. METR observes evidence of useful model judgment on some machine-learning optimization tasks. Those successes leave open the possibility that capabilities transfer further than the weaker judgment results suggest.

The reversal is therefore specific: impressive execution does not automatically validate a system’s assessment of its own work. The models can improve while that assessment remains unreliable. For autonomous research, the relationship between those capabilities matters as much as either score alone.

Better Tests Are Becoming Harder to Build

The expanded evaluations expose a measurement problem: realistic feedback is often expensive, slow, or dependent on experts.

METR introduces lifecycle cost as a rough estimate of the effort required to create, run, and score an evaluation repeatedly. The measure includes development and quality assurance alongside execution and assessment. It aims to capture constraints that human task duration alone misses.

Two tasks can require similar amounts of human work while imposing different evaluation burdens. One might finish with an automatically checked answer. Another might require an expert to inspect reasoning, assess missing context, or judge whether a research contribution matters.

Other burdens include complex environments, computing requirements, monitoring, and long feedback cycles. Tests also require protection against reward hacking, behavior that exploits a scoring rule without accomplishing its intended purpose. Stronger models can reveal shortcuts that earlier evaluations did not anticipate.

In METR’s dataset, tasks with higher estimated lifecycle costs tend to be harder for models after accounting for human task duration. The association suggests that cheap evaluations can miss relevant dimensions of difficulty. It does not establish that expense itself causes weaker performance.

METR estimates lifecycle costs using a model-based judge calibrated against its researchers’ assessments. Those estimates reflect the organization’s experience and constraints. A frontier laboratory might face different costs, and changing model capabilities can alter the economics of evaluation.

The report also finds similar improvement rates across higher-cost and lower-cost subsets of its tasks. That qualification matters. Worse current performance on expensive evaluations does not show that progress on those evaluations has stalled.

The human reference measurements introduce another limitation. Direct human baselines support only 9% of the report’s human-time estimates. Other estimates draw on expert assessments, public evidence, and extrapolation.

That changes how confidently readers should interpret the hours attached to difficult tasks. Estimated duration can still provide useful information, but it does not have the same evidentiary basis as observed completion. Systematic estimation errors could affect comparisons between task groups.

The human comparison also includes AI assistance. METR allowed participants to use Sonnet 5 for implementation, partly to support recruitment and accommodate established working habits. The resulting baseline should not be described as unaided human performance.

Hard-to-verify tasks introduce additional subjectivity. Whether a proposed evaluation is valuable depends partly on METR’s research priorities and beliefs about relevant risks. The organization acknowledges that those judgments can be wrong.

The philosophical critique task raises a related issue about whose judgment becomes the reference. Models are assessed against expert ratings, while agreement can depend on the experts and their available time. That makes the result informative without turning it into a comprehensive measure of human judgment.

These caveats reinforce METR’s earlier time horizon limitations: a benchmark score needs context before it becomes a workplace or safety claim. The updated suite broadens the evidence while adding uncertainty. Better coverage and cleaner measurement do not always arrive together.

The Chart Cannot Date a Research Takeoff

Continued exponential improvement does not establish when AI systems will independently accelerate their own development.

The tempting extrapolation starts with the measured doubling time and extends the curve toward progressively longer assignments. Such projections can illustrate the implications of an unchanged trend. They cannot establish that the same tasks, methods, and relationships will remain representative.

Task selection already affects the fitted trend within the November report. METR finds different results when analyzing subsets of the expanded suite. The researchers discuss several possible explanations, including changes in the difficulty profile that appear as models begin passing newer tasks.

METR also introduces the METR Capability Index, a composite measure built using item response theory, a statistical method relating model ability to task difficulty. This offers another way to track performance across the tested systems. It does not eliminate dependence on modeling choices.

On that index, METR does not find strong acceleration or deceleration across the public frontier it tested. It sees uncertain developer-specific changes under some conditions. Those findings are sensitive to methodological decisions, which limits strong claims about changing growth rates.

The distinction between exponential progress and accelerating progress matters here. A measure can double at a steady interval without its doubling interval becoming shorter. A rising curve alone does not demonstrate a newly intensifying feedback loop.

Research automation adds a causal question. Observing better models while laboratories use AI does not reveal how progress would have unfolded without that assistance. METR explicitly notes that the missing comparison prevents a clean attribution of the trend.

There are also limits to what public releases reveal. Public models do not necessarily represent the most capable systems available inside laboratories. Differences in release timing can complicate attempts to detect changes in the underlying development pace.

The judgment findings likewise support competing interpretations. Research taste, the ability to select promising questions and experiments, could constrain autonomous research. Alternatively, sufficiently strong performance on easier-to-verify work might accelerate development while humans continue choosing directions.

MIT Technology Review’s judgment gap analysis describes that disagreement among researchers. Sayash Kapoor views research taste as a potentially important bottleneck. Najoung Kim emphasizes uncertainty about extrapolating the observed improvements.

Those perspectives do not resolve the issue, but they identify what the present evidence leaves open. Current judgment weaknesses are relevant observations. Treating them as permanent barriers requires an additional argument that the report does not provide.

METR’s own position remains cautious about reassurance as well as alarm. It says public evidence cannot rule out drastic AI research acceleration within the next couple of years. That is a statement about unresolved possibilities, rather than a forecast assigning a date or probability.

The METR time horizon update consequently supports neither a countdown to fully autonomous research nor a guarantee of prolonged human control. It documents substantial progress within a changing evaluation framework. The next inference requires evidence about the research process itself.

Three Signals Matter More Than the Next Viral Curve

The next useful evidence should test whether stronger execution produces better research choices, trustworthy review, and sustained results.

The first signal is adoption of agent-designed evaluations or research proposals after independent expert review. METR’s evaluation-creation task provides a concrete reference point. Future results become more persuasive if experts use the outputs because they answer valuable questions.

The assessment should distinguish a technically correct artifact from a contribution that changes research decisions. Additional reviewers and broader task coverage would help test whether usefulness extends beyond one organization’s preferences. Repeated success would weaken the argument that research taste remains a major constraint.

Continued improvement without adoption would support a more limited interpretation. Agents might become better at producing plausible evaluations while remaining weak at identifying informative ones. That would preserve the gap between polished execution and useful direction.

The second signal is whether AI reviewers become better calibrated on work from equally strong or stronger models. Calibration means that a reviewer’s assessments reliably correspond to actual quality. The November findings give this issue a concrete experimental basis.

A useful follow-up would compare model ratings with independent expert assessments across a broader range of artifacts. Better agreement on difficult outputs would strengthen the case for delegating more review. Persistent overrating would weaken claims that stronger AI authors can be supervised simply by adding AI judges.

This question matters even if agents never conduct an entire research project independently. Teams already need to decide which outputs warrant attention and which require revision. Review that sounds confident while missing defects can move rather than remove the human workload.

The third signal is broader evaluation with stronger human references and delayed outcome checks. More direct baselines would improve confidence in estimated task durations. Assessing what happened after an output was used would test whether short-term scores captured lasting value.

For laboratory reports, the relevant distinction is between doing more activity and producing better research. Additional code or experiments can support progress, but their value depends on the questions they address. Evidence linking agent contributions to validated outcomes would strengthen the autonomous-research case.

These are observation priorities for the coming months, rather than announced publication commitments. A new model release alone will not settle them. The important change would be stronger evidence connecting capability measurements with the decisions and outcomes that the measurements represent.

For teams using AI in technical work, the practical response is to preserve that connection. Keep the original assignment, acceptance criteria, output, and expert assessment available together. A searchable knowledge base can support later comparison when an apparently successful result encounters new requirements.

Read the METR time horizon update alongside the tasks and limitations behind its curve. Before applying its trajectory to your work, identify what counts as success and who can verify it. Separate assignments with immediate tests from decisions whose consequences appear later. Then ask whether your evidence measures completed work, useful work, or merely a convincing output. Those distinctions will remain valuable even as the models improve. The next headline-worthy score deserves attention, but so does a quieter result: an agent choosing a worthwhile question, producing a sound answer, and surviving an independent review. Which part of that chain has your current evaluation actually tested?

Get your work moving in remio

See what needs your attention and take the next step in Workspace.

remio currently supports Windows 10+ (x64) and Macs with Apple silicon.

Your AI Partner at Work
Get more done with remio

Plan. Create. Deliver.
All in one place.

bottom of page