Amazon AWS AgentCore Finds Failures That Healthy Dashboards Miss

Amazon AWS has released an AgentCore optimization capability that finds incorrect agent behavior even when 99% of sessions appear to complete successfully. That conflict matters because operational success does not guarantee that an AI agent fulfilled the user’s request. A workflow can return without an error while skipping an approval, inventing financial data, or failing to update an order.

The new insights capability analyzes production traces across sessions, groups related failures, explains probable causes, and ranks patterns by the number of affected sessions. AWS introduced it on July 23, 2026, as part of Amazon Bedrock AgentCore optimization. The announcement shifts the reliability debate from whether an agent stayed online to whether it produced the intended outcome.

That creates pressure for every company deploying autonomous software, including teams using competing agent frameworks and independent observability platforms. Conventional dashboards remain useful for latency, token consumption, and service errors. However, a green dashboard can conceal behavior that is technically valid and practically wrong.

Amazon AWS Moves Beyond Green Health Checks

The important change is not another trace viewer. Amazon AWS is aggregating traces into ranked explanations of recurring behavioral failures.

Traditional application monitoring begins with explicit signals. A service returns an error code, latency crosses a threshold, or an infrastructure component becomes unavailable. Engineers can connect that signal to a dashboard alert and inspect the affected request.

AI agents complicate that model because they make choices during execution. They interpret requests, select tools, construct parameters, retrieve context, and decide whether a task is complete. Every technical component can operate normally while those choices produce the wrong result.

AWS gives several concrete examples in its failure analysis announcement. An agent might claim that a product is available after an inventory API times out. It might tell a customer that an order changed without executing the modification. It could also skip an approval step and still close the session successfully.

None of those outcomes requires a crashed process. The agent can produce fluent text, report completion, and leave ordinary health metrics unchanged. The failure becomes visible only when a customer complains or someone audits the downstream system.

AgentCore insights attempts to expose that missing signal by examining session traces. A trace is a structured record of the model calls, tool executions, sub-agent activity, and responses within an interaction. The service evaluates each session and identifies where observed behavior departed from instructions or expected task execution.

It currently recognizes 11 failure categories, according to AWS. These include hallucination, incorrect actions, task instruction violations, orchestration problems, and context-handling failures. The analysis focuses on behavioral correctness and policy compliance, instead of waiting for an explicit system error.

Each detected issue receives a trace location, a category, and a natural-language description. AgentCore then clusters related descriptions across sessions. Developers therefore see a recurring pattern instead of a long queue of isolated trace records.

This aggregation changes the unit of investigation. An individual trace answers what happened during one interaction. A cluster indicates whether the same problem repeatedly appears across a meaningful share of production traffic.

AWS also ranks clusters according to their prevalence. A pattern affecting hundreds of sessions appears ahead of an unrelated edge case affecting only a few. That ordering gives engineering teams a defensible basis for deciding which failure to address first.

The distinction matters at production scale. Teams rarely lack telemetry altogether. They lack enough time to interpret thousands of traces and connect similar mistakes before users report them.

AgentCore insights also accepts telemetry from agents outside AgentCore Runtime. Teams can select the CloudWatch log group containing their traces rather than choosing an AgentCore endpoint. This approach expands the feature’s reach beyond applications hosted entirely within Amazon’s managed runtime.

The result is a broader AWS proposition. The company is no longer offering only infrastructure for hosting agents. It is positioning AgentCore as the control layer that observes, evaluates, diagnoses, and improves their production behavior.

Why Silent AI Agent Failures Change the Reliability Test

An agent that returns a successful response has completed a technical transaction, but it has not necessarily completed the user’s task.

That difference exposes a weakness in familiar service-level metrics. Completion rate measures whether a workflow ended. Error rate records recognized failures. Neither metric reliably determines whether the agent selected the correct tool, respected a prerequisite, or changed the intended external state.

Consider a support agent asked to modify an order. It can identify the customer, formulate a reassuring answer, and close the conversation. If it never calls the order-management tool, the session still looks clean unless the team separately checks the business outcome.

The same problem appears in research and analytical agents. A model can fill a missing data point with plausible language instead of invoking an available retrieval tool. The answer may look polished enough to escape casual review. The technical path contains no exception because the model generated exactly what the system allowed it to generate.

AWS demonstrated this issue with a market-trends agent across 10 sessions. AgentCore found fabricated financial claims in 1 session where the agent failed to invoke its data tool. The session completed without an error, even though the behavior violated the system instruction to retrieve real data before presenting numerical claims.

The example is small and comes from AWS, so it should not be treated as an independent production benchmark. Its value lies in illustrating the detection target. The system is looking for a mismatch between the declared workflow and the agent’s actual trajectory.

A trajectory is the sequence of actions and tool calls an agent follows while completing a request. AgentCore’s broader evaluation framework can compare that sequence with an expected trajectory. It can also assess responses against reference answers or natural-language assertions about the intended outcome.

Insights approaches the problem from production behavior rather than a fixed test set. Real customers generate unexpected prompts, combine objectives, omit context, and pursue use cases that designers never anticipated. Those interactions produce failure modes that pre-deployment evaluations may not contain.

This is why the announcement pressures teams that still equate uptime with agent quality. Operational metrics remain necessary, but they address only one layer of reliability. Production agents also require outcome monitoring, behavioral evaluation, and checks against business state.

The risk increases when agents can act. A chatbot’s unsupported answer can mislead a reader. An autonomous workflow can also change records, send communications, approve requests, or initiate transactions. A plausible but incorrect action may have more serious consequences than a visible refusal.

Multi-agent systems introduce another complication. One agent’s output can become another agent’s trusted input. An early fabrication or omitted step can propagate through a workflow without producing a conventional error at any stage.

Teams therefore need to connect three kinds of evidence. Infrastructure telemetry shows whether services operated normally. Behavioral evidence shows whether the agent followed an acceptable process. Business validation shows whether the external outcome matches the user’s request.

AgentCore insights addresses the second layer and can help locate sessions that need validation against the third. It does not eliminate the need for deterministic checks. If a workflow claims to modify an order, the safest design still verifies the resulting order state.

This principle also applies to knowledge work. Teams using agents to summarize research, prepare decisions, or retrieve internal evidence must preserve traceable source material. A searchable engineering knowledge base can make supporting evidence easier to inspect, but the agent’s conclusions still need evaluation.

The standard for production readiness is consequently becoming stricter. The question is no longer, “Did the agent return an answer?” It is, “Did the agent complete the intended task through an acceptable and verifiable process?”

How AgentCore Optimization Turns Traces Into Failure Patterns

AgentCore’s central mechanism is a two-stage analysis that evaluates individual sessions before clustering similar findings across the production workload.

At the first stage, AgentCore examines each session’s messages, reasoning records, tool calls, and final output. It identifies the user’s intent, the agent’s execution strategy, any failure location, and the probable cause. It also classifies problems such as incorrect tool selection, hallucination, or instruction noncompliance.

At the second stage, the service groups similar findings. Failure analysis produces a hierarchy that moves from broad categories to subcategories and then root-cause clusters. Intent and execution analyses produce flatter clusters ranked by frequency.

The hierarchy is important because related symptoms can share one underlying cause. AWS describes a possible top-level cluster called “Agent Bypasses Information Gathering” affecting 116 sessions. Within that group, 114 sessions share a narrower pattern involving skipped prerequisite retrieval. Only 2 represent unrelated edge cases.

That distribution directs attention toward one recurring defect. Fixing the common prerequisite problem should deliver more value than investigating each rare case first. The ranking also reduces the influence of whichever complaint arrived most recently or sounded most urgent.

For root-cause analysis, AgentCore represents a session as an execution graph. Spans within that graph capture inference calls, tool executions, and sub-agent invocations. The system traces backward from the failure and removes unrelated branches before evaluating causality.

AWS says this pruning can narrow a 50-step workflow to the path associated with the bad outcome. The output includes a span identifier, a causality classification, and a recommended fix category. Suggested responses can include revising a system prompt, improving a tool description, or addressing infrastructure.

This mechanism distinguishes pattern analysis from ordinary trace inspection. A trace viewer provides detailed evidence, but an engineer must decide which sessions to open and recognize similarities manually. Insights attempts to perform the first round of that reasoning across the workload.

The capability also generates a map of user intent. It embeds and groups customer requests to show what people are actually trying to accomplish. That view can reveal demand outside the agent’s designed scope or identify a supported task receiving more traffic than expected.

In AWS’s 10-session example, 5 requests involved profile retrieval and portfolio assessment. Three concerned macroeconomic or sector analysis, while 2 requested multi-stock comparisons. Those figures do not establish general usage patterns, but they demonstrate how clustering can guide reliability priorities.

If half of actual requests depend on profile retrieval, that workflow deserves more monitoring than its place in the original product specification might suggest. Intent distribution can therefore influence testing, tool investment, and scope controls.

Execution summaries add another behavioral layer. AgentCore summarizes how each session progressed, then groups similar approaches. Teams can compare the dominant strategy with alternative paths and examine whether particular approaches correlate with failure.

The market agent produced 3 execution patterns in AWS’s example. Six sessions followed a broad portfolio-allocation workflow. Two prioritized profile clarification, while 2 performed comparative stock analysis with sector context.

These views turn telemetry into a behavioral map. Intent clusters show what users request. Execution clusters show how the agent responds. Failure clusters identify where those responses break down.

The system depends on sufficiently detailed telemetry. AgentCore Observability emits metrics, logs, and traces in an OpenTelemetry-compatible format. OpenTelemetry is an open standard for collecting distributed execution data, including the spans needed to reconstruct agent workflows.

AWS’s observability documentation says the telemetry can include session count, latency, duration, token usage, and error rates. Teams can add custom spans, metrics, and logs when the default instrumentation does not capture domain-specific behavior.

Insights can run once for a selected period or on a recurring schedule. Supported recurring frequencies include daily, weekly, and monthly analysis. A one-time run suits post-deployment reviews, complaint investigations, or comparisons around a specific change.

This scheduling model makes the feature retrospective rather than an inline enforcement mechanism. Insights analyzes recorded sessions and produces reports. It does not guarantee that a bad action will be blocked before reaching the user or an external system.

That boundary is central to understanding the product. Pattern discovery improves diagnosis and prioritization. Guardrails, authorization policies, deterministic validation, and human approval remain necessary when an incorrect action carries material risk.

The New Opponent Is Successful Execution With the Wrong Outcome

The primary conflict is not Amazon versus another cloud vendor. It is the appearance of successful execution versus the reality of failed user intent.

That framing explains why AgentCore optimization sits above existing monitoring. Conventional observability excels at detecting infrastructure problems. It can reveal a timeout, a failed credential check, an overloaded service, or a slow model invocation.

Those signals still matter. A tool returning an authentication error needs an operational fix. An agent entering a repeated loop needs trace-level debugging. Excessive token use requires cost and efficiency controls.

However, successful components can combine into a failed workflow. The agent may select an available but inappropriate tool. It can use the correct tool with incomplete parameters. It may ignore a policy written in the prompt because no technical control enforces it.

AWS’s earlier debugging guidance separated production problems into quality, reliability, and efficiency. Dashboards and traces help engineers investigate all three, but they still require someone to identify the relevant session.

Insights adds fleet-level behavioral analysis. Rather than beginning with a known incident, a team can ask the system to discover recurring wrong outcomes across a period. That changes observability from an incident response tool into a source of product-quality signals.

The industry context extends beyond AWS. Observability vendors such as Datadog, Grafana, and Elastic can ingest OpenTelemetry traces from AgentCore. Agent evaluation platforms also score conversations, inspect tool calls, and help teams compare prompts or models.

AgentCore’s advantage is integration. AWS can connect runtime endpoints, CloudWatch logs, evaluations, recommendations, batch tests, and controlled deployments within one managed environment. That can reduce the work required to move from a detected problem to a tested change.

Its openness is also strategically important. AWS says insights can analyze an agent running outside AgentCore Runtime when its traces land in a selected CloudWatch log group. The optimization layer can therefore become an entry point for workloads that are not otherwise hosted by AgentCore.

The deeper competition concerns control over the agent improvement loop. Production telemetry reveals a failure. Analysis identifies a shared cause. A recommendation proposes a prompt or tool-description change. Batch evaluation tests that change, and live traffic can compare versions.

Amazon’s July AgentCore updates describe recommendations, batch evaluations, and A/B testing as parts of this loop. Recommendations use traces and evaluation results to suggest prompt or tool-description changes. Batch tests look for regressions before deployment, while A/B tests compare versions using production traffic.

That integrated loop can attract enterprise teams that do not want to assemble separate systems for hosting, telemetry, evaluation, and experimentation. It also increases dependence on the AWS control plane, even when the underlying agent runs elsewhere.

Independent tools retain room to compete through multi-cloud support, specialized evaluation methods, or closer integration with existing data platforms. Enterprises may also prefer to keep sensitive traces within established observability systems rather than duplicate them in another service.

OpenTelemetry reduces some portability concerns because it standardizes the telemetry format. Yet compatible data does not guarantee equivalent analysis. Failure taxonomies, judge models, clustering methods, and root-cause explanations remain product-specific.

The most meaningful comparison is therefore not a feature checklist. Teams should ask whether an analysis system finds costly behavioral failures earlier, explains them accurately, and connects findings to safe remediation.

A high count of generated findings is not enough. Useful observability must distinguish a widespread product defect from an unusual but harmless execution path. Otherwise, developers receive another queue that demands manual triage.

This is where scope ranking becomes commercially important. An incident affecting a large share of a core user intent deserves faster attention than an equally dramatic failure in a rare unsupported request. AgentCore’s combination of intent and failure clustering attempts to provide that context.

The feature ultimately challenges a comfortable operational assumption. A stable endpoint and low error rate can coexist with an unreliable product. Teams deploying agents must measure correctness at the level where customers experience it.

What Amazon AWS Insights Still Cannot Prove

A generated root-cause explanation is evidence for an investigation, not proof that the system identified the complete or correct cause.

AWS says AgentCore can locate a failure in a trace, classify causality, and recommend a type of fix. Those outputs remain automated judgments over complex, probabilistic behavior. The company has not published independent accuracy measurements for the new insights capability in its announcement.

The examples also come from controlled demonstrations. The market-trends scenario contains only 10 sessions, with one silent hallucination. That is useful for explaining the interface, but it does not show performance across millions of noisy production traces.

Real deployments contain ambiguous outcomes. A user can change goals halfway through a session. Business rules may depend on external context missing from the trace. A correct response can look unusual, while a conventional response may conceal an incorrect downstream state.

Telemetry quality presents another limit. The analysis can only reason over information that instrumentation captures. If a custom tool omits key inputs, outputs, or business identifiers, the trace may not contain enough evidence to determine what happened.

Privacy and security also require careful handling. Session traces can include user messages, retrieved records, tool parameters, and model outputs. Organizations need appropriate access controls, retention settings, redaction, and regional policies before centralizing that data for analysis.

Sampling introduces a tradeoff. Analyzing fewer sessions reduces processing demands but increases the chance of missing rare failures. Analyzing every session improves coverage but can produce more findings, higher operational overhead, and greater exposure of sensitive content.

Frequency ranking can also undervalue low-volume, high-severity events. A repeated minor formatting defect may affect more sessions than one unauthorized financial action. Teams cannot rely on prevalence alone when severity, regulatory exposure, or reversibility differs.

The service’s recommendations deserve similar caution. A prompt change may reduce one failure pattern while creating another. A clearer tool description can improve selection for common cases but distort behavior around edge cases.

AWS’s evaluation system offers a response through ground truth, batch testing, and A/B comparisons. Ground truth provides a known response, expected tool sequence, or behavioral assertion against which a session can be measured. Still, teams must define those references correctly.

LLM-based evaluators bring their own uncertainty. A judge model can misread domain rules or reward a plausible explanation that masks a factual error. Deterministic code-based evaluators remain preferable for exact values, required formats, and verifiable business state.

For example, an evaluator can check whether an agent sounded helpful after changing an order. Only a direct system query can establish whether the order actually changed. High-risk workflows should treat that state check as part of execution, not as optional post-session analysis.

Teams also need human review for emerging clusters. A natural-language label can accelerate understanding, but an engineer or domain owner should inspect representative traces before approving a remediation. The cluster’s name may oversimplify several distinct causes.

The safest interpretation is that insights narrows the search space. It identifies sessions, patterns, and probable causes that deserve attention. It does not transfer accountability from the organization to the analysis service.

This distinction should shape deployment policy. Low-risk content agents can tolerate retrospective discovery and gradual correction. Agents handling payments, access control, medical guidance, or regulated approvals need preventive controls around each consequential action.

The product’s value will depend on how well teams combine those layers. Behavioral analysis can find what ordinary dashboards miss. Deterministic validation and policy enforcement must stop the failures that cannot safely reach production.

What to Watch After the AgentCore Optimization Launch

The next test is whether AWS can turn plausible behavioral analysis into measurable improvements across large, diverse production workloads.

The first signal is independent evidence of detection quality. Customers should look for published case studies that report how many sessions were analyzed, which failure patterns emerged, and how findings compared with expert review. Precision matters because false clusters waste engineering time, while missed clusters preserve the original risk.

Useful reports should also separate prevalence from severity. A platform that ranks only by session count can misdirect teams when rare failures carry greater financial or compliance consequences. Custom severity controls would strengthen the product’s prioritization claim.

The second signal is the performance of the full remediation loop. AWS now connects insights with recommendations, batch evaluations, and A/B testing. Teams need evidence that suggested changes reduce the targeted pattern without lowering task completion elsewhere.

That requires stable version comparisons and representative evaluation sets. A prompt adjustment that improves yesterday’s complaint sample may fail on next week’s traffic. Continuous monitoring should reveal whether improvements persist as user intent changes.

The third signal is competitive and customer adoption outside AgentCore Runtime. AWS allows teams to connect external agents through CloudWatch log groups. Broad usage through that path would suggest the optimization layer has value beyond Amazon’s hosting environment.

Adoption will also reveal whether OpenTelemetry provides enough shared context across frameworks. Agent traces differ in how they record reasoning, tools, memory, and sub-agent activity. Reliable cross-framework analysis requires consistent semantic data, not merely valid trace formatting.

For developers, the immediate action is to compare operational success with business success. Select several high-value user intents and define what completion means in the downstream system. Then verify whether existing telemetry records the evidence needed to evaluate those outcomes.

Teams should also establish a review cadence before turning on recurring reports. Assign owners for the most important failure categories, define severity rules, and require representative trace review before changing prompts or tools.

Knowledge workers evaluating agent output can apply the same discipline. Keep source material available, record which tools the agent used, and verify consequential claims against the underlying evidence. A personal knowledge workflow can preserve context for that review, but it cannot replace judgment.

Amazon AWS has correctly identified a gap that production teams can no longer ignore. An AI agent can stay available, respond quickly, and complete every visible step while still failing its user.

The lasting question is whether organizations will treat behavioral analysis as another dashboard or connect it to enforceable quality controls. Start with one important workflow, compare AgentCore’s clusters with verified outcomes, and measure whether the resulting fixes reduce real customer failures.