OpenAI Audit SWE-Bench Pro Finds Defects in 30 Percent of Tasks

OpenAI conducted an audit of the SWE-Bench Pro coding benchmark. The review found defects in about 30 percent of the evaluated tasks.

Developers use SWE-Bench Pro to measure how well large language models solve real software engineering problems. The audit examined a public subset of 731 tasks. Pass rates for frontier models rose from 23.3 percent to 80.3 percent over eight months.

The increase looked impressive. Yet the audit showed that many tasks contained problems that distorted those results.

Audit reveals widespread task flaws

OpenAI checked test strictness, prompt clarity, test coverage and prompt accuracy. Roughly 30 percent of tasks failed one or more of these checks.

Some tests required outputs stricter than any real-world deployment would demand. Other tasks supplied prompts too vague for models to understand the intended goal. A smaller group of tasks left important functions untested or contained misleading instructions.

These issues mean reported scores do not always reflect genuine model capability.

Why evaluation quality matters now

Model performance on SWE-Bench Pro influences hiring decisions, research priorities and product roadmaps. When tasks contain defects, scores lose reliability.

Companies building agents or coding assistants need to know which numbers they can trust. An inflated pass rate can hide remaining weaknesses in long-horizon reasoning or repository navigation.

The audit therefore serves as a warning rather than a simple scorecard update.

Data quality problems that distort scores

Strict tests formed the largest category of defects. A task might require exact formatting that real users never request. Models lose points even when the functional solution works.

Insufficient prompts created a second category. Developers received only partial descriptions of the bug or feature request. Models guessed at missing constraints and sometimes failed for reasons unrelated to coding skill.

Incomplete test coverage and misleading prompts rounded out the list. In several cases the provided tests missed edge conditions that matter in production code.

OpenAI recommendations for model teams

The report advises teams to review individual task results instead of relying solely on aggregate numbers. Developers should inspect failed cases to separate genuine capability gaps from benchmark artifacts.

OpenAI also noted growing value in using AI agents themselves to audit large evaluation sets. Agents can flag strict tests or ambiguous prompts faster than manual review alone.

The suggestion shifts part of the quality burden from model training to benchmark maintenance.

How quickly reported progress can mislead

The jump from 23.3 percent to 80.3 percent occurred in eight months. That pace suggests rapid capability gains. The audit indicates part of the measured gain stems from task characteristics rather than model architecture alone.

Teams that treat the headline number as ground truth risk overestimating readiness for production software engineering work. Tasks with overly strict or incomplete tests reward narrow pattern matching instead of robust code understanding.

Limits of current coding benchmarks

SWE-Bench Pro attempts to mirror real repositories and issue descriptions. The audit shows the difficulty of creating test cases that stay both realistic and fair across many repositories.

Similar problems appear in other benchmarks that rely on unit tests or hidden test suites. When maintainers update tests without re-auditing the full set, drift accumulates over time.

The pattern suggests periodic third-party audits may become standard practice for high-stakes evaluations.

What remains uncertain after the audit

The report does not name every defective task or quantify how much each flaw changed scores. Public release of the full audit dataset would allow independent verification.

It also leaves open how other popular coding benchmarks compare on the same dimensions. Without that comparison, it remains unclear whether SWE-Bench Pro is an outlier or representative of the broader field.

Signals to watch in coming months

Teams releasing new models will likely publish scores on both the original and any revised task sets. Large drops on the revised set would indicate how much earlier numbers relied on flawed tasks.

Benchmark maintainers may release updated task versions or clearer scoring rubrics. Adoption of those updates by multiple labs would signal growing attention to data quality.

Finally, papers that combine agent-based auditing with human review could appear. Early results from such hybrid methods would show whether the approach scales to other domains.

Developers evaluating coding agents should treat headline benchmark numbers as starting points rather than final proof of readiness. Checking task-level results and watching for revised datasets provide clearer signals than aggregate percentages alone.