AWS's Agent Evaluation Metric for Multi-Turn Conversations Exposes the First Wrong Turn

AWS introduced its Agent Evaluation Metric for multi-turn conversations on September 10, 2026, targeting a failure that final-answer scores routinely conceal. An agent can make one bad decision, carry the resulting state through several turns, and finish with an incorrect answer. A task-level evaluator records one failed conversation. It does not identify where the failure began.

The distinction matters because later turns can appear independently defective when they merely consumed corrupted information. Treating every failed turn as a separate problem sends engineers toward several symptoms instead of one cause. It can also make a model update look worse than it is.

AWS calls its proposed framework AEM. The first published dimension measures correctness through truthfulness and completeness at each response or action turn. The larger contest is between outcome-only scoring and evaluation that preserves the causal structure of an agent’s trajectory.

That contest extends beyond AWS. AgentBench previously evaluated agents across eight interactive environments, while the original tau-bench examined conversations involving users, agents, tools, and domain rules. Both helped shift evaluation away from isolated prompt-response pairs. AEM pushes the argument toward turn-level diagnosis inside each conversation.

AWS Turns One Failed Conversation Into an Error Map

The important change is not another score. It is a method for separating the first mistake from every failure that follows it.

The AEM framework starts with annotated conversations containing expected responses and tool calls. It evaluates each turn against that reference, assigns a pass or fail result, and records a specific reason when a turn fails. Those results are then composed into a conversation-level score.

AWS illustrates the problem with a five-turn sales-report request. During turn two, the agent chooses the relevant action but supplies “profit” when the expected parameter is “revenue.” Turns three through five operate on the incorrect result.

A conventional failure count sees four broken turns. AEM identifies one root cause at turn two and three cascading failures. The later turns receive the label prior_action_failed, indicating that their outputs are wrong because they depend on an earlier error.

This attribution changes the engineering interpretation. Four failures might suggest weaknesses across several prompts, tools, or reasoning steps. One root error points toward a specific argument-selection problem.

AEM evaluates two kinds of turns under the same hierarchy. A response turn contains the text presented to a user. An action turn contains a tool selection and its arguments.

For response turns, completeness asks whether the answer covers everything required by the request. Truthfulness asks whether its claims remain factually consistent with the reference. For action turns, completeness checks whether all required parameter keys are present. Truthfulness checks whether the supplied values are semantically correct.

Action turns also require structural validation. The evaluator must determine whether the agent chose the correct tool and action before judging the fields passed into them. A perfectly formatted argument object does not rescue a call to the wrong API.

The published version treats turn correctness as binary. Each turn passes or fails, although AWS says the same decomposition can support continuous grading of individual claims or fields. The default overall score is the unweighted proportion of passing turns.

That score is only the surface result. The useful material sits underneath it: the failed dimension, affected field, first failure turn, root-cause count, cascade count, and chain length. A dashboard can therefore show that correctness declined while also identifying whether truthfulness or completeness caused the change.

This is the central reversal behind the Agent Evaluation Metric for multi-turn conversations. A lower score does not necessarily mean the agent generated many independent errors. It can mean one early decision contaminated a long dependency chain.

Outcome Scores Hide the Failure That Engineers Need to Fix

An outcome score answers whether the workflow succeeded, while turn attribution answers why it failed. Production teams need both answers.

End-state evaluation remains valuable. A support agent either issued the correct refund, or it did not. A research agent either produced a grounded report, or it did not. A scheduling agent either changed the intended calendar entry, or it changed something else.

The problem begins when that verdict becomes the entire diagnosis. A failed final state can result from a wrong tool, a missing argument, an incorrect value, an incomplete response, or an upstream action whose bad output polluted everything downstream. Those causes demand different fixes.

A tool mismatch can point toward routing instructions or tool descriptions. A missing parameter can expose schema ambiguity. An incorrect value can indicate weak context selection, reasoning, or reference data. An incomplete user response can reveal a presentation failure even when every tool call succeeded.

One holistic score merges those defects. It also gives teams little help when comparing releases. Suppose a new model produces the same task-success rate as the previous version. It may still have traded fewer tool-selection errors for more incomplete responses.

That swap matters in production. Missing a secondary detail in a draft report differs from sending the wrong amount to a financial system. Equal aggregate scores can conceal unequal risk.

The need for layered evaluation already appears across the agent ecosystem. A recent description of evaluation architecture divides testing into runs, traces, and threads. Runs cover individual model or tool operations. Traces cover one complete agent turn, while threads cover multi-turn conversations.

That structure complements AEM’s argument. Conversation-level evaluation reveals whether the user’s objective survived the full interaction. Turn-level evidence reveals the moment and dimension in which behavior diverged.

Earlier benchmarks established why interactive behavior deserves its own evaluation surface. The AgentBench research tested 27 models across eight environments and associated failures with long-term reasoning, decision-making, and instruction following. Those properties emerge through interaction, not from one polished answer.

The tau-bench paper went further by simulating user-agent conversations in retail and airline environments. It evaluated the resulting database state against an annotated goal state and measured consistency across repeated trials. Its original experiments reported that leading function-calling agents completed fewer than half the tasks.

These benchmarks and AEM answer different questions. End-state benchmarks test whether an agent reached the required result under realistic conditions. AEM offers a way to inspect which turn first broke correctness and how the damage propagated.

Neither view should replace the other. An agent might take an unexpected but valid route and still reach the correct state. A rigid trajectory comparison could penalize that flexibility. Conversely, a correct final answer might conceal an unsafe or unstable route that happened to recover.

The practical response is layered scoring. Teams can preserve outcome checks for release decisions, then use turn-level dimensions and traces for diagnosis. Strict action ordering should apply only where sequence affects correctness or safety.

This also changes who feels pressure from the AWS proposal. Evaluation vendors and internal platform teams must move beyond a single success percentage. Agent developers must maintain richer reference data. Product owners must decide which dimensions deserve separate gates instead of accepting one blended quality number.

How the Agent Evaluation Metric for Multi-Turn Conversations Finds the First Break

AEM works by comparing each turn within its full trajectory, assigning a typed failure, and preserving dependencies between actions.

The process starts with a golden dataset, meaning a reviewed set of conversations that defines expected behavior. Each example needs more than a final answer. It should include correct response content, expected tools, required parameters, valid values, and dependencies between turns.

AWS recommends human annotation, or human review when a stronger model helps bootstrap the references. This requirement is substantial. A decomposable evaluator cannot produce meaningful diagnoses when the underlying gold record is vague or wrong.

The evaluator first establishes the turn type. A response turn is judged for coverage and factual consistency. An action turn is judged for tool selection, required keys, and semantically correct values.

AEM uses semantic comparison where exact string matching would be too brittle. “NYC” and “New York City” can represent the same value. “Third quarter revenue figures for 2024” can match “Q3 2024 revenue” without sharing an identical string.

AWS’s conceptual example places a semantic scorer behind a configurable threshold, with exact matching as a fast path. A score above the threshold passes. A score below it produces a truthfulness failure.

Threshold selection becomes a product decision rather than a universal constant. A strict evaluator generates false failures when harmless wording differs. A loose one accepts values that sound related but change the task’s meaning.

A calendar assistant might treat “tomorrow afternoon” as a range that requires clarification. A reporting system might need an exact fiscal period. A compliance workflow may require literal identifiers. One semantic threshold cannot express every domain’s risk tolerance.

Completeness has similar context dependence. Optional parameters should not become failures merely because the golden trajectory used them. Required fields must be distinguished from convenient ones. Otherwise, the evaluator rewards imitation of the reference rather than successful execution.

The failure taxonomy makes those judgments inspectable. AWS lists categories for tool or action mismatches, missing or extra parameters, inconsistent parameter values, incomplete responses, and inconsistent responses. Each label maps to a structural check or one of the correctness sub-metrics.

Then dependency attribution separates original faults from inherited ones. A turn receives prior_action_failed only when it would have passed with correct upstream information. That condition is important. A later turn can contain a new independent error even after an earlier failure.

Consider a research agent that retrieves the wrong document in turn two. It then correctly summarizes that document in turn three. Turn three is wrong relative to the user’s goal, but its local transformation may be valid. AEM should mark the retrieval decision as the root cause and the summary as inherited failure.

Now suppose turn four invents a statistic absent from the retrieved document. That hallucination is not merely inherited. It introduces another root cause, even though the trajectory was already corrupted.

Reliable attribution therefore requires explicit dependency logic. Simply labeling every turn after the first error as a cascade would undercount independent failures. AEM’s value depends on whether evaluators can distinguish inherited state from fresh mistakes.

The framework also records action-chain length. AWS groups chains into single-call, two-step, and complex sequences of at least three steps. Longer chains create more opportunities for an early defect to influence later work, making root attribution more useful.

Once calculated, the structured output can feed dashboards and regression checks. Teams can compare model versions by overall success rate, correctness dimension, root-cause type, and chain length. A release can then fail because tool mismatches increased, even when a blended score barely moved.

AWS presents the method as framework-agnostic, while also showing integration with the Strands Agents evaluation SDK. The custom evaluator docs describe the surrounding evaluation system that can collect traces and run additional evaluators.

This portability matters. The proposal is more useful as a measurement pattern than as an AWS-specific feature. The core sequence remains stable: define dimensions, score each turn, attribute dependencies, compose the results, and monitor changes.

The Real Contest Is Diagnosis Versus Flexible Agent Behavior

The more precisely an evaluator defines a correct path, the greater the risk that it punishes a valid alternative.

Agents differ from deterministic workflows because they can reach the same outcome through several acceptable routes. One agent may retrieve a customer record before checking policy. Another may inspect policy first, then retrieve the record only when needed. Both paths can be valid.

A golden trajectory can accidentally turn one successful example into the only accepted behavior. This problem becomes sharper when evaluators compare action order, selected fields, or intermediate wording. A diagnostic framework needs structure, but too much rigidity converts evaluation into imitation testing.

AWS addresses part of this risk by allowing semantic comparisons and order-invariant steps. Teams can mark actions whose order does not matter, so alternative sequences receive credit. That approach helps, but it does not eliminate the underlying design problem.

The golden dataset must encode invariants, not every incidental choice made by an annotator. Required outcomes, prohibited actions, essential parameters, and state transitions are stronger targets than a single preferred transcript. They describe what correctness demands while leaving room for legitimate variation.

This is where outcome-only scoring retains leverage. Database state, generated artifacts, and verified external effects can reveal success without prescribing a path. A turn-level evaluator should explain failures around those checks, not displace them.

The Agent Evaluation Metric for multi-turn conversations also begins with a deliberately narrow definition of correctness. Truthfulness and completeness do not cover safety, instruction retention, planning quality, efficiency, user satisfaction, or recovery behavior.

An agent can pass every truthfulness check while exposing confidential data. It can provide a complete answer after making unnecessary high-risk calls. It can also obey the immediate request while forgetting a constraint established five turns earlier.

AWS describes correctness as the first dimension in an extensible pattern. Future work is expected to apply the method to safety, with multilingual and multimodal evaluation planned later. Until those dimensions arrive and undergo validation, AEM should not be treated as a complete measure of agent quality.

Automated judges add another uncertainty. Semantic scoring may rely on embedding models, learned scorers, or LLM judges. Each can introduce threshold sensitivity, domain blind spots, and version drift.

A judge can also disagree with human reviewers for principled reasons. Domain experts may know that two similar phrases carry different operational meanings. In finance, medicine, or compliance, a superficially equivalent value can change the permitted action.

AWS recommends correlating decomposed scores with human or gold labels for costly errors. Pearson or Spearman correlation can show whether automated scores track reviewer judgments. That validation should occur for each sub-metric rather than only for the final composite.

Human review remains necessary around ambiguous and consequential cases. The goal is not to automate every judgment. It is to route attention toward the conversations where a human decision has the highest value.

The broader agent-building guide similarly recommends establishing evaluation baselines before optimizing model choice. It also treats human intervention and layered guardrails as parts of reliable deployment.

AEM can make those baselines more informative. It cannot decide which errors an organization can tolerate. A truthful but incomplete answer and a complete but false answer both fail correctness, yet their business consequences can differ sharply.

The unweighted mean introduces the same issue. It assumes every passing turn contributes equally. Production owners may eventually need weights for risky actions, critical fields, or irreversible state changes.

Teams should resist compressing the decomposed evidence too quickly. A single composite number is useful for trend detection, but release decisions should still inspect the underlying failure mix. Decomposition creates value only when people retain it.

AEM Changes the Regression Conversation

Turn-level attribution makes model comparisons actionable because it connects a quality decline to a specific failure class and location.

Agent teams regularly change prompts, models, tool schemas, retrieval logic, memory systems, and policies. Any modification can improve one part of a workflow while damaging another. A final success rate often lacks enough resolution to explain the trade.

Suppose a smaller model maintains the same overall conversation score but generates more missing parameters during three-step chains. That pattern suggests the apparent parity may not survive more complex production requests. Engineers can isolate those chains before broad deployment.

Another release might reduce factual response errors while increasing action mismatches. Product owners then face a real choice. A better writer is not necessarily a safer operator when it selects the wrong tool more often.

AEM’s named sub-metrics create stable comparison points. Truthfulness can be tracked separately from completeness. Root causes can be grouped by tool, action, field, conversation length, or model version.

The framework also supports operational triage. If many failed turns share one upstream cause, teams can prioritize the first failing action. Fixing that action can remove several downstream failures at once.

This is more efficient than reading every red trace as an independent incident. It also produces clearer ownership. A schema team can investigate missing parameters, while a retrieval team examines incorrect source values.

For knowledge-intensive agents, trace diagnosis should include the information available when each action occurred. Teams need versioned prompts, retrieved passages, tool responses, and conversation state. Without that record, an evaluator may locate a failed turn without revealing why the model chose it.

That requirement connects evaluation to knowledge management. A searchable engineering knowledge base can help teams preserve specifications, incident findings, and evaluation decisions beside their test evidence.

Production cases should continually expand the golden dataset. A surprising user request, tool failure, or ambiguous correction can become a reviewed regression case. This keeps evaluation aligned with real behavior instead of a static laboratory script.

Teams should also store successful alternative paths. Failed traces show what must be prevented, while diverse successful traces reveal how much flexibility the evaluator should allow. Both are necessary for avoiding brittle trajectory rules.

The greatest organizational change may be in release reviews. Instead of asking whether the new agent scored higher, reviewers can ask which dimensions improved, where new root causes appeared, and whether longer chains became less reliable.

That conversation is harder to summarize in one dashboard tile. It is also closer to the decisions teams actually need to make.

Three Signals Will Show Whether AEM Travels Beyond AWS

AEM becomes consequential only if teams can reproduce its attribution, calibrate its judges, and extend it without losing comparability.

The first signal is public validation of root-cause labels against human-reviewed trajectories. AWS’s worked example clearly explains the mechanism, but the post does not report internal Amazon Quick Suite production numbers. The next useful evidence would measure agreement on first-failure turns and cascade labels across varied domains.

High agreement would strengthen the claim that AEM shortens debugging. Frequent disagreement would expose dependency attribution as the framework’s weakest link. Teams should watch for evaluations that separate simple linear chains from branching workflows, retries, and recovery attempts.

The second signal is adoption outside one framework. AWS provides a Strands Agents integration, yet the methodology is described as portable. Implementations in other tracing and evaluation systems would test whether its taxonomy survives different representations of turns, tool calls, and state.

Cross-framework adoption would also encourage shared definitions. If every platform interprets truthfulness, completeness, and inherited failure differently, scores will remain local. Common schemas and reference cases would make comparisons more credible.

The third signal is the promised expansion beyond correctness. Safety will be the most important test because safe behavior cannot always be represented as another factual field comparison. A dangerous action can use correct parameters, follow the user’s request, and still violate policy.

A successful safety extension would show that the decompose-evaluate-compose method handles qualitatively different dimensions. A weak extension would suggest that AEM is best understood as a focused correctness debugger, not a general agent-quality metric.

Developers should not wait for that roadmap before improving their tests. Start by choosing several consequential conversations and annotating the outcomes, required actions, critical fields, and dependency structure. Run the agent repeatedly, then compare the first genuine error with the later turns that inherited it.

Keep final-state checks beside the turn-level verdicts. Review semantically ambiguous cases with domain experts. Record valid alternate paths so the evaluator does not mistake flexibility for failure.

The Agent Evaluation Metric for multi-turn conversations makes a persuasive case for changing the unit of diagnosis. Its lasting value will depend on whether independent teams can agree on what broke first. The next question for any agent team is concrete: when your dashboard reports a failed conversation, can it identify the decision that actually caused it?