IBM Agent Consistency Tests Reveal Why One Success Is Not Enough

IBM agent consistency tests have exposed a sharp conflict between benchmark success and dependable behavior. A GPT-4.1 agent completed 77.4% of its AppWorld runs, yet cleared every one of five attempts on only 53.0% of tasks.

That 24.4-point gap changes what a successful agent demonstration means. One correct run can show that an agent has the required capability. It cannot show that users will receive the same result tomorrow, or even five minutes later.

IBM Research is addressing that distinction with new consistency guidelines in ALTK-Evolve. The system finds unstable decisions in a recorded agent trajectory, then converts those weaknesses into reusable instructions. Its early results suggest that reliability gains do not always require a larger model.

The work also puts pressure on the standard way vendors present agent performance. A leaderboard average can reward an agent that works frequently while concealing how often it fails after previously succeeding.

IBM Agent Consistency Results Expose the Missing Metric

IBM's central finding is that average accuracy and repeatable success describe two materially different products.

The researchers tested a ReAct agent powered by GPT-4.1 on 168 tasks from AppWorld's test_normal split. ReAct is an agent design that alternates reasoning steps with actions, such as calling an application or searching stored information.

Each task received five fresh runs. According to the IBM consistency study, the baseline agent achieved a Mean@5 score of 77.4%. Mean@5 averages the success rate across five attempts.

That figure sounds suitable for a strong product demonstration. However, the agent achieved a Pass^5 score of only 53.0%. Pass^5 counts a task as successful only when all five attempts succeed.

The difference between those results is the consistency gap. Here, it reached 24.4 percentage points. On the hardest task group, IBM says the gap reached 30 points.

This distinction matters because Pass^5 is not the familiar Pass@5 metric used in some coding evaluations. Pass@5 asks whether at least one of five attempts succeeds. Pass^5 asks whether every attempt succeeds.

Those metrics support different products. Pass@5 works when a system can generate several candidates, test them, and keep the winner. Pass^5 fits workflows where each execution must be dependable.

A developer asking an agent for five possible code patches can benefit from one valid answer. An accounts team asking an agent to reconcile a transaction cannot safely accept four correct runs and one damaging mistake.

The same issue applies to contract analysis. An agent that identifies an obligation during one review but misses it during another is not merely variable. It creates inconsistent business decisions from identical evidence.

IBM's results do not show that GPT-4.1 lacks the ability to perform these tasks. A 77.4% average establishes that it often can. The results show that capability did not reliably survive repetition inside this particular agent setup.

That is the article's central reversal. The first success is evidence of possibility, not evidence of operational reliability.

The broader evaluation community has reached a similar conclusion. Anthropic's agent evaluation guidance separates Pass@k from Pass^k and recommends matching the metric to the product's requirements.

Anthropic offers a simple illustration. A task with a 75% per-trial success rate has roughly a 42% chance of succeeding across three independent trials. The apparent quality drops because the requirement changes from occasional success to uninterrupted success.

Averages still have value. They help compare overall capability and reveal whether a change improves results across a task set. They become misleading when buyers interpret them as a reliability guarantee.

For enterprise deployments, both numbers belong in the evaluation. Mean@k tells teams how often an agent works. Pass^k tells them how many tasks remain dependable across repeated use.

Why an Agent Can Change Course at Temperature Zero

An agent can become inconsistent even when the prompt, model, tools, and decoding temperature appear unchanged.

An LLM chooses each next token from a probability distribution. Some decisions have a clear favorite, while others contain several nearly tied options.

IBM describes these as sharp and flat decisions. A sharp distribution places most probability on one choice. Small computational changes are unlikely to alter the winner.

A flat distribution spreads probability across several plausible choices. A tiny numerical shift can change which token comes first, sending the agent onto a different path.

That difference becomes important in tool-using systems. A single altered token can change an API selection, search query, tool argument, retry decision, or interpretation of an intermediate result.

An ordinary chatbot can absorb some wording variation without changing its final meaning. An agent acts on its intermediate choices. Each altered choice changes the information available at the next step.

The risk compounds across a long trajectory. If an agent makes dozens of decisions, several may sit near unstable boundaries. Only one needs to flip for a later tool call or final answer to fail.

IBM ran its ReAct agent at temperature 0.0. The researchers therefore argue that ordinary sampling did not cause the observed variability.

Temperature zero does not make every hosted model execution mathematically identical. Request batching, hardware behavior, floating-point operations, and provider-side implementation details can slightly alter probabilities.

Those shifts often remain invisible in ordinary text generation. They matter when two choices are nearly tied. A slight change can reorder them, despite an unchanged user request.

A fixed seed also has limits. It controls part of the generation process, but it does not freeze every layer of a remote inference system. Nor does it make an uncertain decision more decisive.

This mechanism explains why a successful rehearsal can fail during a live demonstration. The agent has not necessarily forgotten the task. It has entered a different branch of the same decision tree.

Consider an agent asked to count completed activities in a note. It might count every checkbox marker during one run. During another, it might also count a legend that explains the checkbox format.

Both paths can initially look reasonable. Only one produces the intended total. The error arises from an unresolved decision about how to interpret the document, not from missing access to the note.

Longer workflows magnify this behavior. A research agent might choose a different source, misunderstand an ambiguous date, or stop after the first matching document. Each choice shapes everything downstream.

That makes AI agent reliability partly an orchestration problem. A better base model can raise the chance of making good decisions, but it does not guarantee that its borderline choices disappear.

External conditions add another layer. ReliabilityBench evaluates repeated execution alongside paraphrased requests and tool failures. Its production stress framework treats consistency, robustness, and fault tolerance as separate dimensions.

IBM's current experiment isolates repeated runs more narrowly. That focus makes the consistency gap easier to inspect, but it does not represent every production failure an agent will encounter.

Real deployments face changed data, expired credentials, API rate limits, partial responses, and evolving schemas. Stable behavior under identical input is therefore a starting requirement, not the entire reliability standard.

ALTK-Evolve Turns Uncertainty Into Targeted Guidance

ALTK-Evolve tries to repair unstable decisions without replaying every task against a live environment.

The new process begins with one recorded trajectory. A trajectory is the ordered record of prompts, decisions, tool calls, observations, and responses produced during an agent run.

IBM's Consistency Analyzer revisits each decision point using the context already stored in that trace. It requests several alternative completions and measures how much the model's choice varies.

The default configuration draws five completions through one additional model call per decision step. It does not repeat the external tool calls or rerun the complete task.

That design matters in production. A full replay might send another email, modify a record twice, duplicate a purchase, or encounter data that has already changed.

Offline resampling avoids those side effects. It also removes the need for a ground-truth label during the diagnostic stage.

The analyzer is black-box, meaning it does not require model logits or internal access. Teams can apply it to a hosted model if they possess the original trace and can resubmit its decision contexts.

Each decision receives a consistency score. Steps with divergent completions become candidates for additional guidance.

ALTK-Evolve then turns those candidates into concise behavioral instructions. Its broader purpose is to extract useful lessons from earlier trajectories and retrieve them during later tasks.

For the note-counting example, the generated guidance recommended a line-anchored regular expression instead of a basic substring count. It also instructed the agent to inspect multiple search matches before selecting a note.

These instructions address general failure patterns. They do not simply save the correct numerical answer from the original task.

That difference is essential. Memorizing one answer would improve a benchmark item without strengthening the agent. A reusable rule can help with new notes, different checkbox formats, or related search decisions.

The ALTK-Evolve repository now includes the Consistency Analyzer and guideline-generation components. The project uses an Apache 2.0 license and supports integration through the Model Context Protocol.

Its retrieval layer brings relevant guidelines into the agent's context at inference time. This gives the model targeted reminders without requiring every historical trace to remain in the active prompt.

The approach resembles a focused form of operational memory. Instead of asking the agent to reread everything it has done, the system distills recurring lessons and retrieves those associated with the current task.

That distinction also matters for knowledge work. A larger archive does not automatically produce better decisions. Useful memory requires selection, conflict handling, and context appropriate to the active request.

The same principle underpins a well-structured AI knowledge base. Stored material becomes valuable when a system can retrieve the right evidence without flooding the working context.

ALTK-Evolve's method is narrower than a general knowledge platform. It targets agent behavior and uses trajectory-derived guidelines. Still, both cases expose the same constraint: retaining information is easier than applying the right information at the right moment.

The mechanism also creates a feedback loop. Teams can collect traces, detect unstable steps, generate candidate guidance, and evaluate whether that guidance improves later runs.

However, the system does not eliminate the need for tests. A generated rule can be wrong, overly broad, or harmful outside the situation that produced it.

Teams still need versioning and rollback. They also need to track which guideline affected a decision, especially when an agent performs regulated or financially consequential work.

The Reliability Gain Is Large, but the Evidence Is Narrow

IBM reports a substantial improvement, although the study remains an early evaluation on one primary benchmark configuration.

After adding consistency guidelines, Pass^5 increased from 53.0% to 69.0% on the 168 AppWorld tasks. That represents a 16-point improvement in tasks passed across all five runs.

Mean@5 also rose, moving from 77.4% to 81.0%. The consistency gap therefore narrowed from 24.4 points to 12.0 points.

This result matters because the average did not decline. A method that improved repeatability by forcing the agent into a consistently mediocre strategy would not solve the underlying problem.

IBM says the guidelines maintained or improved Mean@5 at every difficulty level. The medium group gained 22.9 points in Pass^5, while the hard group gained 14.3 points.

The relative gains were 44% for medium tasks and 45% for hard tasks. Easy tasks gained 12.2 points, with less room available for improvement.

IBM also tested guidelines on a different but related task within the same AppWorld scenario. Pass^5 increased by 13 points, compared with the 16-point same-task gain.

That result supports the claim that some guidelines transfer beyond one recorded trajectory. It does not establish broad generalization across unrelated applications, industries, or agent architectures.

A second experiment used gpt-oss-120b. Its same-task Pass^5 increased from 10.1% to 16.1%, while similar-task performance gained 8.7 points.

The weaker baseline shows that guidance cannot substitute fully for capability. A six-point improvement is meaningful, yet an all-runs success rate of 16.1% remains unsuitable for consequential automation.

The full technical report provides the methodology behind these claims. Even so, readers should treat the results as evidence from the authors' evaluation, not as independent confirmation across production systems.

The main test uses one ReAct configuration, one principal benchmark split, and five repetitions per task. AppWorld offers realistic multi-application scenarios, but a benchmark remains a controlled environment.

Five runs reveal instability that one run hides. They do not precisely estimate rare failures that might emerge across thousands of customer interactions.

The evaluation also raises a statistical question. Pass^k naturally falls as k increases because every additional trial creates another opportunity to fail.

A product team must therefore choose k based on actual exposure. Five successful runs might be a reasonable development screen. It does not prove reliability at enterprise scale.

Guidelines can also become stale. APIs change, policies evolve, and earlier workarounds can conflict with new system behavior.

A rule derived from one unstable branch might suppress a valid alternative elsewhere. The more guidelines a system accumulates, the more important conflict resolution and retrieval quality become.

There is also a difference between consistent behavior and correct behavior. An agent that repeats the same wrong action has perfect behavioral stability and zero task value.

IBM's experiment guards against that issue by reporting Mean@5 alongside Pass^5. Production teams should preserve the same pairing and add outcome-specific safety measures.

For sensitive workflows, consistent correctness must be the target. Stability alone should never become a substitute for factual accuracy, policy compliance, authorization checks, or human review.

Bigger Models Now Face a Systems-Level Competitor

The new evidence challenges the assumption that reliability problems should be solved first by replacing the underlying model.

Model upgrades remain attractive because they can improve many tasks at once. They also require less custom diagnosis than rebuilding an agent's memory or evaluation system.

Yet a larger model does not reveal where an existing agent becomes unstable. It can raise average performance while leaving a meaningful gap between occasional and repeated success.

IBM's approach represents a competing route. Instead of changing the model, it changes the information supplied at high-risk decision points.

The comparison is not strictly model versus memory. Strong systems will use capable models, well-designed tools, targeted context, deterministic checks, and repeated evaluations together.

The pressure falls on agent vendors and enterprise buyers who still report a single success rate. Once Pass^k enters procurement discussions, a high average becomes only part of the evidence.

Buyers can ask whether the same task was repeated, whether the environment was reset, and whether every run reached an acceptable outcome. They can also request results by difficulty instead of one aggregate number.

Developers face a related change. A bug report stating that an agent failed once may no longer reproduce on demand. Teams need complete traces to locate the earlier decision that diverged.

Observability becomes part of product quality. Without stored prompts, tool arguments, results, and intermediate choices, an unstable run can disappear without explaining itself.

That makes evaluation architecture important before deployment. Teams need representative tasks, controlled starting states, explicit success criteria, and enough repetitions to expose variance.

They also need to separate retryable work from one-shot actions. Generating a draft can tolerate multiple attempts. Sending a payment, deleting a file, or approving a contract requires stricter controls.

Pass@k fits the first group when outputs can be verified automatically. Pass^k is more informative for the second group, particularly when users expect the first result to be safe.

Some workflows should not rely on either metric alone. A transaction agent also needs permission boundaries, idempotency controls, audit logs, and deterministic validation before executing an irreversible action.

Guidelines can reduce uncertain reasoning, but they do not replace those safeguards. A reliable agent is a system property, not a model score.

The results also strengthen the case for curated working context. Teams already use knowledge blending to connect stored evidence with active work. Agent guidelines apply a similar idea to behavioral lessons.

The key is restraint. More retrieved context can create distraction, contradictions, and added latency. A targeted instruction has value only when it reaches the correct task and does not displace essential evidence.

This is where ALTK-Evolve's long-term challenge lies. Its diagnostic can identify variable decisions, but the surrounding system must manage a growing collection of lessons.

A successful deployment will need expiration policies, conflict detection, provenance, and evaluations for guideline changes. Otherwise, yesterday's fix can become tomorrow's hidden failure source.

What Will Show Whether IBM's Approach Holds Up

Three signals will determine whether consistency guidelines become a standard agent layer or remain a promising benchmark technique.

The first signal is independent reproduction across other benchmarks and agent architectures. Researchers should test the method on browser agents, coding agents, customer-service systems, and workflows with real API failures.

A repeat of the 16-point Pass^5 improvement would strengthen IBM's central claim. Smaller or inconsistent gains would suggest that AppWorld contains failure patterns especially suited to guideline repair.

Comparisons should include several base models. They should also report token use, latency, analyzer cost, retrieval overhead, and the number of guidelines injected per task.

The second signal is performance over longer time periods. A useful learning system must handle new guidance without accumulating conflicting or obsolete rules.

Teams should measure whether the benefit survives after hundreds or thousands of trajectories. They should also test whether guideline retrieval remains precise as the store grows.

Watch for evaluations that freeze one guideline set and test it against later application versions. Strong performance would show that the rules capture durable behaviors. Rapid decay would expose a maintenance burden.

The third signal is adoption of repeated-run reporting. Vendor scorecards should publish Mean@k and Pass^k together, with k stated clearly.

That shift would let buyers distinguish models that succeed frequently from systems that succeed dependably. It would also discourage demonstrations built around one favorable execution.

The field should go further by combining repetition with paraphrases and controlled tool failures. Identical prompts represent only one kind of production pressure.

IBM's work makes a persuasive case that one successful run is too weak a standard. It also offers a practical method for locating uncertainty without repeating every real-world action.

The remaining question is whether those gains persist outside the research setup. Developers can answer it by rerunning their own important workflows, preserving full traces, and measuring all-run success.

If your agent handles consequential work, do not ask only whether it completed the task. Ask how often it completes the same task correctly, which decisions vary, and what happens when yesterday's guidance meets tomorrow's conditions.