Niklaus Harness Leaderboard Tests Kimi Code, Hermes, Pi, Claude Code, and Codex Beyond Brand Rankings

Joël Niklaus compared nine coding-agent harnesses across 28,849 trials, challenging the idea that choosing a model settles the performance question. The Niklaus harness leaderboard evaluates the software that directs a model’s work while controlling the underlying model and reasoning settings. Its initial results put several recognizable tools near the same accuracy range, with differences in efficiency and repeatability complicating any winner.

The comparison includes Kimi Code, Hermes Agent, Pi, Claude Code, Codex, OpenCode, mini-swe-agent, Qwen Code, and OpenClaw. A harness is the software that manages an AI model’s instructions, tools, and execution loop. Here, that surrounding software becomes the variable under examination.

The central tension is between a product’s reputation and the behavior of a specific tested configuration. These results do not establish that Claude Code or Codex performs universally better with its default model. They examine what happens when different harnesses operate selected open models under controlled conditions, a narrower question with practical consequences for developers.

What the Niklaus Harness Leaderboard Actually Compares

The experiment isolates the surrounding agent software instead of treating every coding product as an inseparable package.

In his release description, Niklaus says the comparison holds the model, provider, and reasoning effort fixed while changing the harness. Each task receives three attempts. The initial evaluation covers GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Flint, using Terminal-Bench 2.1 and DeepSWE.

That design matters because ordinary product comparisons often change several variables simultaneously. A developer switching coding assistants can change the model, instructions, available tools, permissions, and execution behavior together. A performance difference then becomes difficult to attribute.

The release description reports mini-swe-agent at a 51.4% average solve rate across the three models, compared with OpenClaw at 27.3%. Niklaus also says four other harnesses are statistically tied with the leading result. Those figures describe the initial release, rather than a permanent ranking.

His model-specific findings complicate that overall average. According to Niklaus, mini-swe-agent leads on GLM-5.3-Flash, Hermes Agent on DeepSeek-V4.1-Flash, and Kimi Code on Flint. The ordering therefore depends on which model a developer intends to run.

The immediate implication is limited but consequential. A harness that works well with one model does not automatically extract the same relative performance from another. An overall score can help identify candidates, while concealing the pairing that matters for an actual deployment.

Terminal-Bench evaluates agents performing tasks in terminal environments, where software executes commands and interacts with files. DeepSWE supplies a separate software-engineering evaluation. Using both broadens the comparison, although combining results does not make the task collections representative of every development workflow.

The experiment also needs a date attached to its conclusions. When checked for this article, the live application displayed version v0.1.2. Its current description refers to equivalent providers being checked, rather than simply repeating the initial release’s fixed-provider wording.

That distinction deserves attention because a benchmark can evolve after its first announcement. New harness versions, additional trials, or revised controls can change the meaning of a screenshot. Launch figures remain evidence about the launch configuration, even when the current board presents a different ordering.

For readers encountering the original Reddit discussion, the essential correction is therefore about scope. This is a comparison of selected harness-and-model combinations. It is not a comprehensive test of every feature, subscription experience, or default configuration associated with the listed product names.

Coding-Agent Vendors Face Pressure to Explain the Whole Configuration

The results put pressure on claims that attribute an agent’s performance entirely to its underlying model.

For harness developers, the comparison makes implementation choices more visible. Instructions, tool interfaces, retry behavior, and stopping decisions all shape the work a model performs. A strong model can still encounter friction from the software responsible for directing it.

For model providers, the same experiment introduces an attribution problem. A disappointing result inside a particular agent does not identify the model as the sole cause. Conversely, a strong result does not establish that every application using those weights will reproduce it.

For enterprise buyers, this changes what a meaningful evaluation needs to describe. Naming a model and an assistant is insufficient when configuration choices influence outcomes. Buyers need enough information to understand the tested combination and reproduce the behavior relevant to their work.

A hypothetical maintenance task makes the distinction concrete. An agent might need to inspect a repository, identify a failing test, modify a function, and verify the change. Each stage depends on how the harness exposes information and manages the sequence of actions.

If the agent stops after editing without running the relevant test, a plausible patch remains unverified. If it keeps issuing unproductive commands, completion becomes slower and more resource intensive. Those are operational differences, even when the underlying model remains unchanged.

This does not establish which implementation caused any particular leaderboard result. That requires inspecting execution records and testing an intervention. It explains why the controlled comparison asks a useful question that a model ranking alone cannot answer.

The launch findings also put a boundary around the familiar Claude Code vs Codex debate. These product names often stand for complete experiences built around particular model families. Testing their harnesses with selected open models asks a different question from comparing those complete experiences.

A low position in this experiment would not establish that a product’s default service performs poorly. A high position would not establish superiority in code review, interactive collaboration, or organization-specific workflows. Those claims need evaluations that measure those activities directly.

The Kimi Code benchmark result carries the same limitation. Its reported lead with Flint supports investigating that pairing under the tested conditions. It does not establish a general lead across unrelated models or repositories.

For vendors, the constructive response is to publish configurations and explain changes that affect outcomes. Buyers can then distinguish a model upgrade from a harness improvement. That makes performance claims easier to assess and failures easier to investigate.

The pressure is especially relevant for teams operating open models through interchangeable agent software. Their purchasing decision includes the execution layer. They need evidence about what that layer contributes, rather than relying entirely on a recognizable product name.

Similar Accuracy Can Conceal Different Operating Behavior

A close accuracy result leaves room for meaningful differences in resource use, latency, and consistency.

The current board separates several measurements instead of presenting accuracy as a complete answer. Its evaluation method describes external measurement of model usage and distinguishes model processing time from time spent running the agent. Those distinctions help explain why similar solve rates can produce different operating experiences.

The board also plots accuracy against cost and identifies a Pareto frontier, the set of options no competitor beats on both displayed measures. This gives readers a way to inspect competing advantages without assuming that the highest accuracy must always be preferable.

That framing matters when an organization has a fixed workload and limited resources. A small accuracy advantage can be valuable for a difficult task requiring minimal supervision. The same advantage can be less attractive if it requires substantially more execution time across routine maintenance work.

An efficiency result also needs interpretation. Lower spending does not necessarily mean an agent reasons more effectively. It might reflect fewer attempts, earlier termination, shorter outputs, or a different pattern of tool use. The execution record helps distinguish productive restraint from premature failure.

Likewise, longer execution does not establish waste. An agent might spend additional effort checking a patch or recovering from a failed approach. The relevant question is whether the extra work improves an outcome the team actually values.

This is why reporting resource use per attempt and per successful outcome answers different questions. A cheap failed attempt remains a failure. A more demanding successful attempt can still be the better choice when human recovery would otherwise dominate the workflow.

These are evaluation considerations, rather than explanations established for every harness on the board. A chart can reveal a pattern worth investigating. It cannot, by itself, identify the instruction or tool behavior responsible for that pattern.

The mechanism is straightforward in principle. An agent receives information, chooses an action, observes the result, and decides what to do next. The harness controls important parts of that loop, including what tools are available and how their responses reach the model.

Consider a hypothetical repository with a failing build caused by an environment issue. An agent that repeatedly edits application code can consume resources without addressing the cause. Another configuration might recognize the environment problem and redirect its investigation.

The difference would concern task execution rather than raw model knowledge. Establishing that explanation would require matching the observed behavior to the result. Without that inspection, saying that a particular harness is “smarter” goes beyond the evidence.

The Niklaus harness leaderboard therefore supports a more specific comparison than a single winner label suggests. Developers can ask which combinations achieve acceptable accuracy while using resources in a way their workload can support. The answer can differ between interactive development and unattended operation.

It also changes how to read a Kimi Code benchmark headline. A model-specific lead is a useful signal for further evaluation. It remains incomplete without information about failed attempts, execution duration, and the behavior that produced the score.

For the same reason, Claude Code vs Codex comparisons should preserve the tested context. The names provide orientation, but the configuration determines what the result actually describes.

Repeatable Success Is a Different Result From Occasional Success

An agent that solves a task intermittently presents a different deployment proposition from one that solves it consistently.

Niklaus subsequently highlighted reliability across repeated attempts. According to his reliability breakdown, mini-swe-agent solved 51.4% of tasks per attempt on average. It solved 68.6% at least once across three attempts, but 33.0% on every attempt.

Those measurements describe different operational questions. The average captures performance across attempts. Success at least once captures whether repeated opportunities eventually produce a solution. Success every time captures consistency within the measured attempts.

The distinction becomes important when someone watches the agent work. A developer can inspect a failed attempt, change an instruction, or decide that another run is worthwhile. Human supervision can make intermittent success useful, although the time spent supervising remains part of the workflow.

An unattended process faces a different constraint. It cannot assume that a successful alternative attempt exists unless the surrounding system actually runs alternatives and recognizes their quality. Repeated attempts need a selection method, an execution budget, and a way to handle unresolved failures.

This makes an occasional-success score a poor substitute for a consistency measurement. It can show that a configuration has access to a solution under some conditions. It does not establish that the configuration will deliver that solution reliably on demand.

Niklaus reports that five harnesses solved between 32.0% and 33.3% of tasks on every attempt in the cited comparison. He describes those results as too close to call. That supports resisting precise winner claims inside the leading group.

The reliability figures also need limits attached. Three attempts provide evidence about repeatability within that experiment. They do not establish a production failure rate across changing repositories, dependencies, permissions, and user instructions.

Nor does a failure on a benchmark automatically translate into a harmful production action. The evaluator measures whether the task meets its success criteria. Operational consequences depend on what the agent can access and what approval or verification surrounds its work.

For engineering managers, the implication is to separate capability from dependable delivery. A configuration can be capable enough to produce a correct patch sometimes while remaining unsuitable for an unattended workflow. That gap calls for additional verification rather than a stronger marketing label.

A hypothetical dependency update illustrates the difference. An agent might successfully update the package and pass tests during one run, then miss a related configuration change during another. The occasional success proves less about unattended readiness than the headline percentage might suggest.

A team assessing that workflow would care about reproducible completion and clear failure handling. It would also care whether the resulting change is understandable to a reviewer. Those dimensions remain distinct from whether the benchmark verifier accepts the final state.

The leaderboard’s repeated-attempt view brings that distinction into the public comparison. It encourages readers to ask what happens when the agent must work again, rather than treating one successful trajectory as a complete description of the product.

Benchmark Versions and Public Records Limit the Winner Claims

The published results are inspectable evidence, but they remain bounded by benchmark versions, configurations, and the completeness of verification.

Terminal-Bench itself supplies a clear example of why version labels matter. Its 2.1 revision fixed issues in 28 of the previous edition’s 89 tasks. The maintainers reported changes in representative agent-model scores after those repairs.

That history makes a benchmark version part of the claim. A score from Terminal-Bench 2.0 cannot simply be treated as interchangeable with one from 2.1. The tasks and evaluation conditions have changed, even when the displayed metric keeps the same name.

The same principle applies to the harness. A result belongs to the tested software version and settings. If an update changes instructions, tool execution, or termination behavior, the earlier score describes the earlier implementation.

This creates a challenge for fast-moving public discussions. A chart can circulate after the board changes, while comments continue treating the chart as current. Readers need the original configuration and measurement date to determine what the image still establishes.

The project’s public run records offer a route toward checking those details. Availability of records is a transparency benefit. Their publication does not mean every result has already received independent reproduction or complete manual review.

Independent reproduction would require executing the specified configurations and checking whether the relevant patterns persist. Reading the records can support an audit of reported calculations or individual behaviors. Those activities provide different kinds of evidence.

The causal question remains particularly demanding. A score difference can identify an association between configuration and outcome under controlled conditions. Explaining the difference requires examining failures and testing whether a proposed change alters them.

For example, a suspected timeout problem should be tested by changing the timeout while preserving other relevant conditions. A suspected information-loss problem requires inspecting what information disappeared and whether preserving it improves results. Neither explanation follows automatically from a low rank.

Benchmark familiarity introduces another concern worth investigating without assuming misconduct. A system optimized against public tasks can improve on that task distribution without improving equally elsewhere. Evidence from unfamiliar tasks would help establish how broadly the gain transfers.

Task coverage presents a separate limitation. Repository maintenance includes organizational conventions, ambiguous requests, and human review expectations that a fixed benchmark cannot fully capture. A benchmark can still be useful while leaving those deployment questions unresolved.

The practical response is to keep the evidence proportional to the decision. A public result can justify a shortlist or a targeted trial. Replacing a team’s development workflow requires additional evidence from work resembling that team’s actual responsibilities.

Keeping configurations, failure examples, and review decisions together makes those comparisons easier to revisit. A searchable knowledge base can help preserve that evaluation history alongside the technical documents used during testing.

The verification gap does not erase the experiment’s value. It defines what the results currently support and what additional work would strengthen them. That is a more useful reading than either accepting every rank literally or dismissing the comparison entirely.

The Next Tests Should Track Updates, Reproduction, and Unattended Work

The strongest follow-up evidence will connect version changes to reproducible improvements on work beyond the initial comparison.

The first signal to watch is updated harness performance with clear version history. When a developer changes the execution software, a comparable rerun can show whether accuracy, efficiency, or consistency changes. Preserving the earlier configuration makes that comparison meaningful.

An improvement across repeated measurements would strengthen the case that harness engineering materially affects outcomes. A result that disappears under comparable reruns would weaken confidence in the claimed gain. Either outcome supplies more information than an updated winner badge.

This also gives vendors a constructive way to respond. They can identify a failure pattern, explain the relevant implementation change, and publish evidence about the outcome. Readers can then assess a concrete engineering claim rather than infer quality from a product’s reputation.

The second signal is independent reproduction with matching controls. A separate evaluator should document the model, provider conditions, reasoning settings, task version, and harness version. Differences need to remain visible rather than being folded into an apparently identical score.

If independent results preserve the broad ordering or model-specific patterns, confidence in the comparison increases. If results diverge, that points toward conditions that require closer examination. Provider behavior, execution settings, and measurement details become research questions rather than convenient explanations.

Reproduction does not require every number to match exactly. The relevant issue is whether the conclusion survives reasonable repeated measurement. Close results should remain close unless additional evidence supports distinguishing them.

The third signal is performance on unfamiliar work with explicit unattended-success criteria. That means checking whether the agent completes the requested task, verifies the result, and handles failure appropriately. Human intervention should be recorded because it changes what “autonomous completion” means.

Consistent improvements on that work would strengthen the practical importance of the leaderboard’s findings. Gains confined to familiar benchmark tasks would narrow their relevance. The distinction matters more for deployment than another small movement in an overall average.

For teams choosing software now, the evidence supports comparison at the level of the actual pairing. Kimi Code, Hermes Agent, Pi, Claude Code, Codex, and mini-swe-agent deserve evaluation in the configurations a team intends to use. Their names alone cannot settle the outcome.

The Niklaus harness leaderboard gives that evaluation a public starting point, with launch results, repeated attempts, and accessible records. Its lasting contribution will depend on whether those measurements help explain and improve agent behavior.

Before changing your coding workflow, compare a shortlisted pairing against the setup you already trust using representative maintenance work. Record the exact configuration, completion evidence, reviewer effort, and failures that require intervention. Include tasks where a plausible patch is insufficient without a successful verification step. Revisit the comparison after a meaningful harness update, preserving the earlier results so changes remain understandable. Treat the public board as evidence for that decision, while giving your own workflow the final say. The question worth answering is concrete: which tested combination delivers work your team can verify, repeat, and accept with the supervision you can actually provide?