Amazon AgentCore Evaluations Confront the Failure Traditional Monitoring Misses

Amazon published a nine-stage framework for Amazon AgentCore evaluations after finding that healthy infrastructure can conceal deteriorating agent answers. Latency can remain normal. Error rates can stay flat. The agent still responds, but its confident answer may cite the wrong figure or customer.

That conflict matters because traditional monitoring mainly detects software failures that leave visible technical evidence. Autonomous agents introduce semantic failures, meaning the application runs while its reasoning, tool choices, or conclusions become less reliable.

AWS built the framework around an internal multi-agent system used by sellers and sales leaders. Specialized agents answer performance, customer, and structured data questions through a shared conversational experience. One degraded tool or sub-agent can contaminate the final answer without crashing the application.

The framework represents a shift from evaluating an agent before launch to evaluating its behavior continuously. It connects development experiments, deployment gates, production canaries, adversarial checks, drift detection, observability, and automated recommendations.

The real opponent is not another cloud provider. It is the familiar assumption that uptime, latency, and successful requests provide enough evidence of agent health. Amazon’s case argues that those signals describe availability, not correctness.

Amazon AgentCore Evaluations Turn Quality Into a Release Condition

The significant change is that AWS treats answer quality as an operational control, not a final testing exercise.

AWS published the framework on October 16, 2026, based on its work with an internal production system. The system includes Performance Insights, Customer Insights, and Data Insights agents serving different business questions.

The Performance Insights Agent handles territory-scoped revenue and performance questions. The Customer Insights Agent combines product usage, support, contract, and engagement signals. The Data Insights Agent converts natural-language questions into SQL queries over structured datasets.

These agents operate behind one interface, but their failure surfaces remain distinct. A customer-resolution tool can choose the wrong subsidiary. A retrieval component can return weak context. An orchestrator can route a request correctly yet distort the sub-agents’ findings during synthesis.

AWS therefore evaluates the system at three levels. Tool-level checks examine capabilities such as retrieval, entity resolution, and SQL generation. Agent-level checks score each specialized agent from request to response. Multi-agent checks assess routing, context transfer, and final synthesis.

Each level gets its own evaluation dataset, thresholds, and promotion criteria. This structure matters because a favorable end-to-end average can hide a failing component. The failure may affect only one customer segment, query type, or orchestration path.

The framework also measures five dimensions throughout the lifecycle. These are safety, quality, performance, cost, and operational control. Quality receives the most detailed treatment, but AWS says safety remains a shipping requirement.

The nine stages form a closed loop: define, experiment, merge, gate, canary, adversary, drift, pulse, and improve. The stages stretch from the first commit to production monitoring and subsequent revisions.

AWS connects them through AgentCore Evaluations, which supports controlled and live assessment modes. On-demand evaluation runs against selected datasets, traces, or sessions. Online evaluation continuously samples production interactions.

That continuity is the important product claim. A team can apply the same evaluators before deployment and after users encounter the agent. The evaluation standard should not change merely because the software crossed a production boundary.

AWS also combines AgentCore with CloudWatch, S3, Redshift, Lambda, and Amazon Quick. Those services handle traces, evaluator output, persistence, custom scoring, alarms, and dashboards.

This is not a new foundation model or agent architecture. It is an operating model for deciding whether an agent remains fit for use. That narrower change could prove more consequential for enterprises moving beyond demonstrations.

The Nine-Stage Framework Starts Before the First Test Run

Amazon’s framework begins by defining failure in business terms, because an undefined quality target makes every later gate arbitrary.

Stage one establishes the evaluation contract. Teams define business outcomes, evaluation metrics, failure thresholds, safety policies, latency budgets, cost targets, and operational responses.

For AWS sellers, success means receiving current and accurate account intelligence at the correct level of detail. Leadership users need reliable weekly and cross-business review inputs without extensive manual preparation.

Those outcomes are then translated into measurable failure categories. AWS identifies metric swaps, wrong periods, wrong entities, unsupported figures, incorrect symbols, and unauthorized calculations.

This taxonomy is more actionable than one general accuracy score. A falling pass rate says that something changed. A category-level result indicates whether the problem sits in retrieval, entity resolution, time interpretation, formatting, or generated reasoning.

AWS combines several evaluator types because no single scoring method covers every risk. Built-in evaluators address relevance, faithfulness, helpfulness, instruction following, and tool selection. Customized evaluators target domain-specific errors.

Deterministic checks compare outputs with an exact expected result. They suit account identifiers, contract dates, revenue values, SQL results, and structured formats. LLM-as-a-judge evaluation uses a model and rubric when several responses could reasonably satisfy the request.

That distinction is essential. Asking another model to verify an exact identifier adds uncertainty where a simple comparison would work better. Using exact string matching to score nuanced reasoning creates the opposite problem.

AWS also uses third-party evaluator libraries and a golden set, meaning a versioned collection of representative queries with expected behavior. That set includes production-derived examples, failures from previous releases, and constructed edge cases.

At the tool level, the golden set tests capabilities such as customer identification. At the agent level, AWS covers comprehensive, reporting, reasoning, and ambiguous questions. Production failures can become permanent regression cases.

Stage two compares proposed changes against the same evaluation contract. Prompt revisions, model replacements, tool additions, and dataset updates all run through on-demand evaluation before merging.

This comparison spans quality, latency, cost, and token consumption. A candidate that raises accuracy while sharply increasing response time and cost does not automatically win. The framework forces those tradeoffs into the decision record.

AWS reports that three months of targeted iteration raised its pass rate from 93 percent to 96 percent. It also says pure hallucinations declined from 138 to fewer than 52 per 10,000 responses.

Unauthorized calculations reportedly fell from 59 to fewer than 44 per 10,000 responses. These figures come from AWS’s internal account and have not received an independent audit.

That qualification is important because the source is both a technical case study and a product argument. The results show what AWS says it achieved in one environment. They do not establish a universal baseline for agent reliability.

Even so, the evaluation design remains useful outside AWS. Teams need a shared definition of acceptable behavior before they can compare prompts, models, retrieval systems, or orchestration changes.

Documenting that contract also creates organizational memory. Teams using a searchable engineering knowledge base can preserve failure definitions, evaluation decisions, and incident evidence alongside implementation notes.

CI Gates and Canaries Attack Different Failure Windows

Pre-release gates stop known regressions, while production canaries search for deterioration that no deployment triggered.

Stage three places a smaller sentinel dataset inside continuous integration. AWS runs this set automatically after staging deployments, including changes to code, prompts, models, tools, or configuration.

The sentinel emphasizes high-risk scenarios across quality, safety, latency, and cost. It runs faster than the full golden set, making it suitable for frequent checks. A breached threshold can block the deployment.

AWS maintains separate sentinels for tools, agents, and multi-agent workflows. A release cannot pass merely because the final agent score looks acceptable. A regression inside a critical tool can independently stop promotion.

The company describes a data-normalization change that affected customer-name resolution. The tool returned unresolved entities for certain formats, and the agent guessed a matching customer. That behavior risked returning accurate data about the wrong account.

A deterministic entity-resolution case caught the problem before production. This example captures why agent testing must look below the final prose. A fluent answer can still be functionally incorrect.

Stage four expands the release check into a full quality gate. AWS evaluates the candidate against the complete golden set and uses frozen tool responses to isolate agent behavior.

This replay design removes live tool availability and changing source data from the comparison. Each candidate receives the same validated tool outputs. Score differences should then reflect the agent’s reasoning rather than environmental variation.

AWS says its end-to-end gate includes 200 held-out questions. A candidate fails if its aggregate score regresses, misses the production baseline, or leaves too many individual questions below threshold.

That last condition prevents averages from hiding concentrated harm. A model might improve common questions while becoming unreliable for one region, customer type, or request format.

Stage five changes the setting. A canary runs in production against live traffic and synthetic cases, looking for failures caused by external dependencies or data changes.

AWS chose 30-minute evaluation windows. It says 15-minute windows generated excessive false alarms, while 60-minute windows delayed confirmation of a real decline.

One failed window remains an operational signal. AWS pages its on-call team only after three consecutive windows breach the threshold. A separate daily run evaluates the entire golden set at release-gate thresholds.

This design tries to balance sensitivity against alarm fatigue. The canary should detect sustained quality loss without training engineers to ignore noisy notifications.

AWS offers an example involving a retrieval index refresh. The refreshed tool returned broadly related passages rather than the precise context required by the question.

Infrastructure metrics looked normal. Latency remained within expectations, and the technical error rate stayed at zero. The agent continued producing polished responses based on weaker evidence.

The canary detected lower grounding and retrieval precision during the first evaluation window. A trace then connected the poor score to the retrieval tool.

This is the central reversal in the framework. A successful request can represent a product failure, while a technically healthy dependency can still lower answer quality.

A separate AWS and Motorway implementation provides another concrete reference. Its production blueprint reports that incorrect search results fell from one in eight queries to one in 50.

Motorway’s agent helps dealers search vehicle inventory through natural language. Its evaluation pipeline checks tool use, reasoning, and output quality across repeated runs. AWS says issue detection fell from hours to minutes.

The two cases are not directly comparable. They involve different users, datasets, failure definitions, and architectures. Together, however, they show why release testing and production evaluation solve separate problems.

Live Evaluation Exposes the Limits of Static Test Sets

A golden set protects against remembered failures, but real users continually produce inputs that its designers did not anticipate.

Stage six addresses adversarial behavior. AWS applies input and output controls to screen requests, protect sensitive data, and detect attempts to manipulate agent instructions.

The framework uses Bedrock Guardrails for content filtering and prompt-attack detection. AWS says these checks cover all traffic synchronously before a response reaches the user.

The threat model includes direct jailbreak attempts and instructions hidden inside retrieved documents. It also covers attempts to extract internal prompts or expose data outside a user’s authorization.

AWS correctly notes that safety is not confined to one stage. Policies are defined at the foundation, tested during integration and deployment, then enforced in production. Stage six makes the adversarial layer explicit.

Stage seven targets drift, meaning a sustained change in behavior relative to the established baseline. Drift can follow data updates, changing user behavior, stale assumptions, or long-tail requests missing from the golden set.

AWS says it samples 10 percent of live traffic for online evaluation. Quality, latency, and cost scores feed a quality-state calculation using the same 30-minute window structure.

One internal example involved customer identification. Synthetic tests covered name variations, subsidiaries, and abbreviations. Live users also supplied ticker symbols, shorthand names, and typographical errors.

Reported accuracy fell from 98 percent during one window to 81 percent, then 80 percent across the next two windows. CloudWatch alerted the on-call team after the rolling threshold was crossed.

AWS says traces exposed ticker and shorthand handling as the main source of failure. The team updated its normalization logic and promoted low-scoring sessions into the golden set.

Accuracy reportedly returned to baseline in the next evaluation cycle. Again, this is an AWS-reported operational result rather than an independently reproduced finding.

Stage seven also supports controlled exposure to live traffic. After a candidate clears the offline gate, a gateway routes a portion of requests between the current and proposed versions.

The candidate must match or beat the production version across the chosen dimensions. Offline evaluation tests known cases, while live comparison introduces unfamiliar wording and behavior.

This process resembles established software canaries and model monitoring, but agents expand the object being measured. Teams must evaluate trajectories, tool calls, retrieved context, final answers, and session outcomes.

That wider surface creates a measurement problem. LLM-based judges can disagree, inherit model biases, or reward stylistic traits that do not reflect business correctness. Rubrics can also become outdated as products and users change.

Sampling introduces another limitation. Evaluating 10 percent of traffic reduces processing expense, but rare and harmful failures might fall outside the sample. Aggregate scores can also conceal low-volume user groups.

Privacy and data governance require equal attention. Production traces can contain prompts, retrieved documents, tool outputs, account identifiers, and other sensitive context. Retention and access controls must match the data involved.

The framework recognizes safety and operational control, but implementation quality still depends on each organization. AgentCore supplies evaluation infrastructure. It cannot define acceptable behavior or trustworthy ground truth on the customer’s behalf.

What the Amazon Framework Still Cannot Prove

Amazon’s case supports continuous evaluation, but it does not prove that automated evaluators can fully measure trust.

The first limitation is evidence independence. The principal framework, implementation, and performance results come from AWS. The company operates the system and sells the supporting platform.

That relationship does not invalidate the results. It does mean readers should treat reported improvements as case-study findings, not neutral benchmarks against competing evaluation stacks.

The second limitation concerns evaluator validity. A score is useful only when it correlates with the behavior that users actually value. A precise dashboard can create false confidence if its rubric measures the wrong outcome.

AWS tries to reduce this risk by starting with business goals and combining deterministic, customized, and model-based evaluators. Teams must still validate each judge against expert human review.

Inter-rater agreement would be a valuable signal. If evaluators, domain experts, and users frequently disagree, the automated score should not control deployments alone. AWS’s post does not provide that agreement data.

The third limitation is coverage. The framework tests known failure categories and incorporates new production cases over time. It cannot guarantee detection of every novel behavior before harm occurs.

A golden set also risks becoming optimized indirectly. Repeated prompt and model changes may improve scores on familiar examples without delivering broader reliability. Held-out cases and periodically refreshed datasets can reduce that risk.

The fourth limitation is cost. Continuous evaluation adds model calls, trace storage, data processing, dashboards, and engineering work. AWS tracks cost per query and cost per successful response, but publishes no complete implementation total.

A smaller team may not need all nine stages immediately. It could begin with a domain-specific failure taxonomy, deterministic tests for critical facts, a release gate, and sampled production review.

The fifth limitation is remediation authority. Stage nine can generate system-prompt and tool-description recommendations from failed traces. Recommendations are not the same as safe autonomous changes.

A suggested prompt revision can fix one failure while introducing another. It should return through the same experiment, merge, gate, and canary sequence before broad deployment.

Automated rollback also needs careful boundaries. Reverting a prompt or model is relatively contained. Reversing an agent action in a business system can involve payments, messages, records, or permissions.

These concerns separate evaluation from governance. Evaluation estimates whether the agent behaved well. Governance determines what the agent may do, which evidence must be retained, and where human approval remains mandatory.

AWS’s broader AgentOps guidance places evaluation alongside security, versioned artifacts, observability, and human review. That framing is more credible than treating one score as a trust certificate.

Trust is not a property that AgentCore can attach to an agent. It is an accumulated judgment based on outcomes, controls, incident handling, and transparency. Evaluation contributes evidence to that judgment.

For buyers, the right question is therefore not whether a platform offers evaluations. The harder question is whether its measurements cover the organization’s actual failure costs and remain credible under change.

Three Signals Will Show Whether the Closed Loop Works

The framework’s value will depend on evaluator credibility, production adoption, and evidence that feedback improves unfamiliar cases.

The first signal is independent validation of automated evaluators. Enterprises should watch for published comparisons between AgentCore scores, domain-expert judgments, and real user outcomes.

High agreement would strengthen AWS’s claim that evaluations can govern releases. Persistent disagreement would weaken automated gates and increase the need for human review.

The comparison should cover more than general helpfulness. It should test entity resolution, authorization boundaries, tool selection, grounded calculations, session outcomes, and multi-agent synthesis.

The second signal is adoption beyond AWS-authored examples. Production reports from regulated industries, large software vendors, and smaller engineering teams would reveal how portable the nine-stage model is.

Useful reports should disclose dataset construction, sampling rates, false alarms, missed incidents, remediation time, and evaluation overhead. A percentage improvement without those details provides limited guidance.

AgentCore’s online configuration can continuously process traces from an agent endpoint or selected CloudWatch log groups. The online evaluation controls also support configurable evaluators, sampling, and execution status.

The important adoption metric is not the number of enabled configurations. It is whether teams detect semantic failures before users report them and resolve those failures without creating new regressions.

The third signal is closed-loop performance on unseen traffic. AWS says its recommendation system can analyze traces and suggest revised prompts or clearer tool descriptions.

In one reported example, hardcoded values inside few-shot examples appeared in unrelated answers. AWS replaced those values with placeholders and observed fewer hallucinations and stronger grounding.

The next test is whether similar recommendations improve held-out and live interactions, not only the traces that generated them. Every recommendation should pass through the original evaluation loop.

Stage eight supplies the operational foundation for that test. AgentCore Observability records structured execution data through OpenTelemetry, an open standard for traces, metrics, and logs.

AWS says its dashboards connect an alarm to the relevant agent trace, including tool order, inputs, outputs, latency, and token use. Evaluator results follow a separate pipeline into CloudWatch and longer-term storage.

The company reports that mean detection time fell from days or weeks to under four hours. It also targets automated detection before user complaints become the main incident signal.

Those results capture the framework’s most defensible promise. Amazon AgentCore evaluations do not guarantee that an autonomous system will always answer correctly. They make silent deterioration observable enough to investigate, block, or reverse.

Developers should start by identifying the wrong answer that would cause the greatest harm. Enterprise buyers should ask vendors how that failure is measured before release and detected afterward.

Knowledge workers should remain cautious when an agent sounds certain. Fluent language still provides no assurance that the right tool, customer, period, or source sits underneath the response.

The next generation of agent platforms will compete on more than model access and orchestration. They will compete on whether teams can explain failures, reproduce them, and prevent their return.

If your organization already operates agents, examine the last incident that reached a user. Would latency or error monitoring have detected it? If not, that gap is the starting point for a real evaluation contract.