Real-SWE Benchmark Exposes the Gap Between AI Coding Scores and Enterprise Work

Specific Labs released the Real-SWE benchmark with 640 agent runs, and the highest-ranked configuration resolved only 38.8 percent of its assigned work.

That result creates an uncomfortable contrast with public coding leaderboards. Agents increasingly look capable of handling repository-level issues, yet most Real-SWE attempts failed on private production systems.

The difference is not simply that Real-SWE contains harder programming puzzles. Its tasks require agents to reconstruct business rules, navigate unfamiliar architecture, and coordinate changes across code and connected services.

This distinction matters because companies do not hire software engineers to solve isolated, context-free exercises. They need engineers to preserve existing behavior while changing systems that already process customer data, permissions, billing, and infrastructure.

Real-SWE therefore challenges the promise implied by high public benchmark scores. It asks whether an agent can enter an unknown enterprise codebase and complete economically useful work without relying on familiar public artifacts.

The early answer is sobering. The benchmark offers evidence that coding models can produce credible patches, but it does not support unsupervised deployment across consequential enterprise workflows.

What the Real-SWE Benchmark Actually Changed

Real-SWE shifts the evaluation target from public repository issues to private systems with company-specific rules and operational dependencies.

Specific Labs announced the benchmark on September 12, 2026. The company describes it as an evaluation of frontier models on private production codebases licensed from operating businesses.

The initial release covers ten tasks and eight model-and-harness configurations. Each configuration receives eight independent attempts per task, producing 640 scored rollouts.

That repeated-run design matters. A single successful patch can hide substantial inconsistency, while eight attempts show whether an agent solves the same class of problem reliably.

The published leaderboard reports pass@1, meaning the probability that one attempt resolves the task. Specific Labs averages that result across the eight runs assigned to each task.

Fable 5.1 with Claude Code led the initial ranking at 38.8 percent. GPT-6 Astra with Codex CLI followed at 33.8 percent, while Gemini 3.8 Flash with Gemini CLI reached 31.2 percent.

GLM 5.3 with Claude Code recorded 28.8 percent. Grok 4.6 and Muse Spark 1.3 each reached 23.8 percent with their respective agent systems.

Kimi K3 with Kimi Code reached 18.8 percent. GPT-5.6 Sol with Codex CLI finished at 16.2 percent in this particular evaluation.

These results evaluate combinations, not bare language models. A harness supplies tools, controls the interaction loop, and shapes how a model explores and edits a repository.

Specific Labs explicitly says it used native harnesses intended to reflect current engineering workflows. That choice improves practical relevance but complicates model-to-model comparisons.

A stronger score can reflect model reasoning, harness behavior, tool reliability, or the interaction among all three. The leaderboard cannot cleanly isolate those effects.

The benchmark’s deeper change concerns data exposure. Public coding tests normally use open-source repositories, public issues, and visible solution histories.

Private repositories reduce the chance that a model encountered the relevant patch during training. They also prevent an agent from finding an answer through public search.

According to the published Real-SWE results, each task originated from work associated with a real private codebase. Some instructions were reportedly lifted directly from actual engineering work.

That design makes the evaluation less like an exam built for models. It makes it closer to placing a new contractor inside an unfamiliar software organization.

The distinction also changes what failure means. An unsuccessful patch may reflect weak coding, but it can also reveal poor requirement discovery or incomplete system understanding.

That is precisely the territory where enterprise deployments become risky. A patch can compile, pass obvious tests, and still violate a rule embedded elsewhere in the business.

Why Private Enterprise Code Creates a Different Test

The central challenge is not generating more code. It is discovering which behavior the business already depends on and preserving it across multiple systems.

One Real-SWE example asks an agent to fix invoice taxation. The short description hides several conditions involving business configuration, customer exemptions, geographic rules, and external tax services.

The agent must determine when to use a stored rate and when to calculate tax from a buyer’s destination. It must recognize accounts that collect no tax.

It must also preserve exemptions, record invoice totals correctly, and report rejected addresses without stopping the invoice. Settled transactions must be returned to the tax authority.

European transactions add another requirement involving VAT registrations. Completing the work can involve a TypeScript service, TaxJar environments, and an InfluxDB ledger.

None of these individual operations represents an exotic algorithm. The difficulty comes from coordinating them without overlooking a condition or damaging existing behavior.

That pattern appears throughout enterprise software. A request that sounds local often crosses APIs, databases, background jobs, configuration, tests, and operational tools.

Real-SWE environments can expose services and tools including Docker, Kubernetes, GitHub, PostgreSQL, MySQL, Redis, Slack, email, and customer-support systems.

Each task receives only the services its workflow needs. Even so, an agent must decide which systems contain relevant evidence and which are distractions.

The benchmark reports a median instruction length of 1,742 characters. That is shorter than a detailed implementation specification, leaving agents to infer structure from available code and context.

Reference solutions changed a median of 11 files. The benchmark’s comparison data places that above the six-file medians reported for FrontierCode and DeepSWE.

This gap helps explain why repository navigation matters. A model that finds one obvious implementation point can still miss validation, persistence, configuration, or downstream consumers.

Specific Labs says six of the ten tasks had resolution rates below 15 percent. No tested configuration solved every task.

The failure analysis identifies missed requirements as the most common category. Other observed failures included unverified assumptions, integration errors, regressions, and edits to the wrong file.

Those categories resemble ordinary engineering review comments more than syntax errors. They suggest that agents frequently produce plausible local changes without building a complete model of the system.

Time alone did not separate winners from failures. The benchmark says 70 of 98 rollouts lasting under ten minutes failed, a 71.4 percent failure rate.

Among longer rollouts, 398 of 542 failed, representing 73.4 percent. More runtime therefore did not automatically repair weak exploration or mistaken assumptions.

The short-versus-long comparison should not be treated as proof that additional reasoning never helps. Harder tasks probably generate longer attempts, creating a major confounding factor.

Still, the result weakens a convenient explanation. These agents did not fail merely because every run ended before the model could finish typing a solution.

The more credible mechanism is incomplete context formation. An agent must identify requirements, locate their implementation, trace dependencies, and verify the whole change.

That workflow benefits from durable repository knowledge. Engineering teams already maintain architecture notes, incident history, decisions, and technical documents for the same reason.

A searchable engineering knowledge base can reduce repeated discovery work for humans. Agent systems increasingly need an equivalent context layer.

Real-SWE does not establish that persistent memory would solve these tasks. It does show why stateless code generation is an incomplete model for enterprise engineering.

Public Coding Scores Meet Private-Code Reality

The main contest is between benchmark-visible coding ability and dependable performance inside systems that models have never encountered.

SWE-bench changed coding evaluation by asking models to resolve real GitHub issues within repository snapshots. That was a major improvement over isolated function-generation tests.

The original benchmark connected issue descriptions, repository state, and execution-based grading. It helped move the field toward agents that explore, edit, and test software.

However, those repositories and issue histories are public. As models improve and evaluation data circulates, contamination becomes harder to exclude.

Contamination occurs when a model’s training data contains benchmark tasks, relevant patches, or close copies. A high score can then mix generalization with recognition or memorization.

SWE-bench Verified attempted to improve task quality through human review. Its verified dataset retained 500 samples after screening issue descriptions and tests.

That review itself demonstrated how difficult coding evaluations are to construct. OpenAI reported that 68.3 percent of screened samples were removed for specification, testing, or related problems.

The filtered benchmark remained useful, but it did not eliminate public exposure. Model developers could still study the repositories, tasks, grading behavior, and common failure patterns.

More recent audits increased the pressure for new evaluation designs. OpenAI’s 2026 benchmark audit reported design and contamination concerns in widely used coding evaluations.

The audit also estimated that about 30 percent of SWE-Bench Pro tasks were broken. Reported problems included strict tests, missing requirements, weak coverage, and misleading prompts.

Real-SWE addresses one part of this problem by keeping underlying repositories and solutions private. Models cannot retrieve an exact public patch if that patch was never published.

Yet private data introduces a different tradeoff. Outside researchers cannot freely inspect every repository, reproduce every task, or audit hidden tests.

That reduces transparency precisely when a benchmark makes commercially significant claims. Readers must trust the benchmark operator’s licensing, anonymization, task construction, and scoring process.

Real-SWE provides task examples, aggregate results, confidence intervals, and a failure taxonomy. Those disclosures help, but they do not equal an openly reproducible evaluation.

The ten-task size presents another limitation. Repeated runs measure consistency across attempts, but repetition does not create broader task coverage.

A benchmark with ten tasks can be sensitive to task selection. One architecture, language mix, or business domain can materially influence rankings.

The model-and-harness pairing adds further uncertainty. Claude Code appears with more than one model, while Codex CLI also appears with multiple models.

That variation provides useful deployment information. It does not produce a controlled comparison where every model receives an identical scaffold and tool policy.

The correct interpretation is therefore narrower than a universal model ranking. These scores show how named configurations performed on this Real-SWE release under its documented setup.

They do not prove that the top model is best for every company. They also do not establish that the lowest-ranked configuration lacks useful coding capabilities.

The benchmark instead supplies a warning against transferring public scores directly into private deployment expectations. That warning aligns with the evaluation field’s broader reassessment.

METR makes a related distinction in its task methodology. Its self-contained tasks intentionally give models clear success criteria and limited dependence on organizational history.

METR notes that real work often depends on prior conversations, tacit knowledge, and familiarity with an existing codebase. Its methodology compares agents more closely with low-context contractors.

Real-SWE moves further into that low-context problem. It retains executable grading while introducing company-specific conventions and operational relationships.

This is the benchmark’s most important reversal. Strong performance on public issues no longer serves as sufficient evidence for dependable enterprise autonomy.

The Leaderboard Pressures Buyers as Much as Model Vendors

Real-SWE puts enterprise evaluation teams under pressure because vendor scores cannot substitute for testing inside the buyer’s own environment.

A company evaluating coding agents usually faces an information imbalance. Vendors know their models, while buyers know their repositories, workflows, and risk tolerance.

Public leaderboards give both parties a common reference. They simplify comparison, but that simplicity becomes dangerous when the deployment environment differs from the benchmark.

Real-SWE makes the mismatch visible. Even its strongest configuration failed more than six of every ten attempts across the evaluated task set.

A buyer should not translate that number directly into an expected production failure rate. Ten benchmark tasks cannot represent every repository or engineering organization.

The number still changes the procurement conversation. Teams need evidence about reliability on their own code, not only performance on public issue histories.

That means building internal evaluations from representative, previously completed work. Suitable tasks can include bug fixes, migrations, permission changes, and integration work with known outcomes.

The reference solution should not become the only acceptable implementation. Reviewers must distinguish functional correctness from superficial similarity to an engineer’s original patch.

Hidden tests also need scrutiny. A benchmark can punish a valid alternative if its verifier encodes unstated implementation details.

OpenAI’s benchmark audits show how frequently this happens. Evaluation quality requires experienced engineers to compare instructions, tests, reference changes, and agent traces.

Organizations should also test full configurations. Real-SWE’s decision to score model-and-harness pairs reflects how coding agents operate in practice.

Repository search, shell access, test execution, context management, and retry policies can materially alter performance. Measuring only the underlying model ignores those dependencies.

Security belongs in the same evaluation. Private source access creates questions about retention, provider controls, secret exposure, logging, and permissions.

An agent that solves more tasks can still be unsuitable if it receives broader access than the organization can safely grant. Capability and deployment risk must be assessed together.

Review burden provides another critical measure. A patch that eventually works may require so much human investigation that it offers little productivity benefit.

Teams should record how often an agent misses requirements, creates regressions, or needs corrective prompting. Those measures map closely to Real-SWE’s failure categories.

They should also track variance. One successful demonstration says little about whether the agent can repeat the result on the next attempt.

The Hacker News benchmark discussion captured both sides of this issue. Some participants welcomed repeated pass@1 trials because they reveal consistency.

Others argued that current tools still work best through close human collaboration. Under that view, the skilled operator remains part of the system being evaluated.

This “centaur” model pairs a human with an AI agent. The human supplies judgment, context, and review while the agent handles exploration, drafting, and mechanical changes.

Real-SWE does not compare autonomous agents with expert-agent teams. Its results therefore should not be read as evidence that coding assistants lack practical value.

They indicate something more specific. The tested configurations are not reliably replacing the contextual and verification work performed by experienced engineers.

That difference matters for deployment claims. Assisting an engineer and autonomously owning an enterprise change are separate capability thresholds.

Buyers should demand that vendors state which threshold their evidence supports. A public score alone cannot answer that question.

What the Real-SWE Numbers Do Not Prove

The benchmark provides a meaningful warning, but its small private sample cannot support sweeping conclusions about every model or enterprise codebase.

First, Real-SWE’s tasks come from companies selected by Specific Labs. The published examples include a consumer application, a fintech platform, and enterprise sales software.

Specific Labs says one represented application serves more than 200,000 users. Another platform reportedly processes more than 100,000 bank statements.

Those details establish commercial relevance, but the companies remain anonymous. Readers cannot independently assess their engineering quality, architecture, or domain complexity.

Second, the benchmark’s privacy prevents complete public replication. That privacy protects licensed code and reduces direct contamination, yet it concentrates validation authority.

Independent evaluators need controlled access to confirm task provenance, verifier quality, environment parity, and scoring. Without that review, some uncertainty remains unavoidable.

Third, the leaderboard combines different models with different native harnesses. This resembles actual product usage, but it weakens conclusions about underlying model intelligence.

A configuration can lose because its search strategy fails, its tool calls break, or its context management discards relevant evidence. Those are product failures, but not identical failures.

Fourth, the benchmark uses automated resolution as its primary outcome. Passing a verifier is necessary, though enterprise acceptability can require more.

Production review may consider maintainability, observability, security, migration safety, style, and future operational burden. A binary pass cannot fully capture those dimensions.

Conversely, a valid patch can fail an imperfect verifier. The history of SWE-bench audits shows that hidden tests can reject reasonable solutions or overlook incomplete ones.

Real-SWE says required behavior must be stated or reasonably discoverable. That standard is sensible, but “reasonably discoverable” still requires judgment.

Fifth, anonymous business tasks can favor models or harnesses whose exploration patterns match the benchmark. A different repository structure could reverse some rankings.

The 95 percent confidence intervals shown on the leaderboard acknowledge sampling variation across runs. They do not eliminate uncertainty from selecting only ten tasks.

Sixth, the launch page makes a broad claim that most enterprise tokens remain hidden from frontier models. That statement supports the benchmark’s motivation but lacks public measurement detail.

It should be treated as the company’s framing, not as an independently established statistic. The safer conclusion is that substantial enterprise context remains private.

Finally, low autonomous resolution does not equal low productivity impact. An agent can save time through investigation, test generation, documentation, or draft patches without finishing independently.

The opposite is also true. A completed patch can create review costs or subtle risks that offset its apparent speed.

These limitations do not make Real-SWE irrelevant. They define the conditions under which its findings are useful.

The benchmark is strongest as evidence of a deployment gap. It is weaker as a definitive ranking of models or a forecast of engineering employment.

Specific Labs is also a commercial participant in enterprise data and evaluation. Its company profile describes a business built around realistic enterprise environments and datasets.

That does not invalidate the work. It makes independent replication and transparent methodology more important.

A credible benchmark ecosystem needs multiple operators, rotating private tasks, and third-party audits. No single leaderboard should become the final authority.

Three Signals That Will Determine Whether Real-SWE Matters

Real-SWE will become consequential only if its private-code signal survives expansion, independent review, and repeated model releases.

The first signal is task-set growth. Ten tasks can reveal failure modes, but a larger collection must cover more languages, architectures, and business domains.

The expansion should preserve private provenance while publishing enough metadata for readers to understand task diversity. It should also separate repository-backed samples from other task formats.

If rankings remain similar across a broader test set, the claim of a persistent enterprise gap becomes stronger. Large rank changes would show that selection shaped the launch results.

The second signal is independent evaluation. External researchers or model providers need controlled access to audit tasks, verifiers, and execution environments.

A useful audit would examine whether prompts contain sufficient information, whether hidden tests accept alternative implementations, and whether reference solutions represent genuine production work.

Agreement from independent reviewers would strengthen confidence in the reported resolution rates. Discovered test flaws would narrow or revise the benchmark’s conclusions.

The third signal is progress across repeated releases. Private benchmarks lose value if tasks leak, become training targets, or remain static while agent systems adapt.

Specific Labs will need fresh held-out tasks and clear versioning. Results should distinguish improvements from models, harnesses, retries, and task exposure.

A rapid rise in scores on new private tasks would indicate better generalization. Gains limited to old tasks would suggest optimization around the benchmark itself.

Enterprises should watch failure composition alongside the headline pass rate. Fewer missed requirements and unverified assumptions would matter more than prettier generated patches.

Consistency should improve as well. A model that occasionally resolves a task remains difficult to trust when the same prompt often produces a regression.

Human-agent comparisons would add another useful layer. Measuring review time and total completion time could show where agents already create value without full autonomy.

Real-SWE has placed the right question in front of model developers and buyers. Can an agent safely change software whose history, conventions, and business logic were never public?

Its first results say current systems often cannot. The best configuration resolved fewer than four attempts in ten, while missed requirements dominated the observed failures.

That conclusion should prompt better evaluation, not blanket rejection. Coding agents can remain valuable when engineers control scope, provide context, and verify consequences.

The practical next step is to test representative internal work before widening permissions. Teams should compare configurations, repeat every task, and study why apparently reasonable patches fail.

The Real-SWE benchmark matters if it changes buying behavior from trusting public scores to demanding private evidence. Will your next coding-agent trial measure generated code, or completed work that your business can safely accept?