Senior SWE-Bench Tests AI Agents on Real Senior Engineer Work

Senior SWE-Bench is an open benchmark that measures whether AI agents can complete tasks at the level of experienced software engineers. The benchmark splits work into two categories: adding new features and fixing bugs that require runtime investigation.

Each task comes from real pull requests in repositories that range from single libraries to multi-service systems. Engineers who have made hundreds of commits to those repositories wrote the task instructions.

The first category gives agents natural-language instructions similar to messages from product managers. Verification relies on behavior tests that were generated from expert recipes. The second category supplies logs, profiling data, and other runtime evidence so agents must investigate the root cause before proposing a fix.

Current top scores remain low. Claude Opus 4.8 paired with a high-effort Mini-SWE-Agent reached 24 percent success. Claude Sonnet 5 reached 19.4 percent. GPT-5.5 reached 16 percent. Every leading model still fails more than three-quarters of the tasks.

Tasks Span Many Files and Require Hundreds of Steps

The average feature task touches eleven distinct files. Even the strongest agents need hundreds of individual steps to finish. Instruction length sits at roughly one-third the median length found in the earlier SWE-Bench Pro set, yet difficulty stays high because the work demands coordination across modules, services, and tests.

Developers who review these results note that short prompts do not simplify the underlying complexity. The agent must still locate relevant code, understand existing design choices, write passing tests, and preserve performance characteristics that were not stated in the prompt.

Leaderboard Shows Clear Gaps in Quality and Taste

Success rate alone does not capture the full picture. Reviewers also score whether the final change matches the style and judgment expected from a senior engineer. Even when an agent produces a working patch, many solutions are rejected for poor modularity, missing edge cases, or awkward interfaces with neighboring services.

The gap between raw correctness and acceptable engineering taste is the clearest signal on the board. Teams that have tried these agents inside their own codebases report similar findings: the model can locate the bug or add the requested endpoint, yet the surrounding implementation often needs substantial human cleanup.

Why Senior SWE-Bench Matters for Tool Builders

Most existing agent benchmarks use short, self-contained problems. Senior SWE-Bench forces agents to operate inside real, long-lived repositories where naming conventions, test patterns, and deployment constraints already exist. This setting matches the daily workload of senior engineers far better than isolated coding problems.

Companies building internal agents now use Senior SWE-Bench scores as one filter before wider deployment. A model that cannot clear 25 percent on this benchmark is unlikely to reduce senior-level ticket time in production settings.

Remaining Uncertainties

The benchmark is still young. It is possible that better scaffolding, improved retrieval, or new agent architectures will raise scores quickly. It is also possible that the current ceiling reflects fundamental limits in how language models plan across many files and implicit requirements.

No independent audit of the task set has been published yet. Some observers question whether the expert recipes used to build tests introduce bias toward certain solution styles. Others ask whether the repositories chosen over-represent certain languages or architectures.

What to Watch Next

Three milestones will reveal whether progress is steady or stalled. First, any release that pushes the top score above 35 percent will show whether current methods can scale. Second, the publication of inter-rater agreement numbers on taste judgments will clarify how reliable the quality filter is. Third, the first public case study from a company that integrated a Senior SWE-Bench–qualified agent into its review queue will indicate real productivity impact.

Readers who track these three signals will have a clearer view than leaderboard numbers alone can provide.

Senior SWE-Bench does not claim to measure everything an engineer does. It focuses on two concrete slices of work that already have verifiable outcomes. Those slices are enough to show that current agents still need experienced humans to finish the job.