AWS Agent Monitoring Splits Quality From Infrastructure Failures
AWS has introduced a two-layer agent monitoring pattern for a four-agent airline reservation system, confronting failures that ordinary infrastructure dashboards often miss.
The approach combines Amazon Bedrock AgentCore Evaluations with AWS DevOps Agent. One layer scores an agent’s behavior and output quality. The other investigates the cloud resources supporting that behavior.
That separation matters because an AI agent can remain online while giving irrelevant answers, selecting the wrong tool, or mishandling a multi-step task. Conversely, a sound agent workflow can fail because a downstream service, permission, or network path breaks.
AWS is effectively challenging a familiar operating model. Application teams usually treat latency, errors, logs, and resource health as the main signals of production reliability. Agentic systems require those signals, but they also need evidence about whether the system made the right decisions.
The AWS monitoring pattern turns that distinction into an operational workflow. Continuous evaluation identifies poor sessions, while an autonomous investigation traces suspected infrastructure causes.
The result is not a single all-knowing monitor. It is a division of responsibility between quality judgment and infrastructure diagnosis. That division is the central promise, and also the part enterprises must pressure-test.
AWS Agent Monitoring Now Covers Behavior and Infrastructure
The important change is that AWS agent monitoring treats response quality and service health as separate production signals.
The example uses an airline reservation application built around four agents. A supervisor coordinates specialized agents responsible for flight, user, and reservation tasks.
That structure resembles many emerging enterprise systems. A customer request enters through one interface, but several agents and services may participate before the system returns an answer.
A flight agent might retrieve available routes. A user agent could access traveler information. A reservation agent might complete a booking after the supervisor chooses the next action.
The architecture increases specialization, but it also expands the failure surface. A bad result may begin with reasoning, routing, context, tool selection, permissions, data access, or an unavailable service.
Traditional monitoring can report whether a request completed, how long it took, and which component returned an error. Those measurements remain necessary, but they do not answer whether the completed task was useful.
AgentCore Evaluations addresses that behavioral gap. An evaluation uses defined criteria to score agent interactions, traces, or sessions instead of judging only system availability.
AWS presents online evaluation as the continuous part of the workflow. Teams can apply evaluators to production interactions and inspect score distributions across real sessions.
On-demand evaluation provides a narrower path. Operators can select a session or trace for closer analysis when a score, complaint, or incident deserves investigation.
This distinction gives teams two useful views. Aggregate scoring can reveal a deteriorating pattern, while session-level inspection helps reconstruct the path behind one poor outcome.
The example’s dashboards show quality summaries and distributions rather than a simple pass or fail indicator. That matters because agent quality often erodes gradually.
A model update might reduce instruction adherence without causing application errors. A modified prompt could improve average answers while making one important task less reliable.
The behavioral layer therefore creates a signal that infrastructure monitoring cannot generate by itself. It asks whether the application fulfilled its intended purpose.
AWS DevOps Agent operates on the other side of the boundary. It analyzes operational data and resource relationships to investigate faults in the supporting AWS environment.
The example connects that investigation to Amazon CloudWatch, which collects application and infrastructure telemetry. The DevOps agent can use those signals to construct a topology and trace a suspected failure path.
That workflow reframes monitoring as a sequence. First, detect a meaningful quality problem. Next, determine whether infrastructure contributed. Then, produce evidence and proposed remediation for human review.
The combination does not eliminate existing observability practices. It adds behavioral evaluation above them and an investigative agent across them.
For platform teams, this changes the minimum production standard. A green service dashboard no longer means the agent completed its actual job.
Four Agents Create More Failure Paths Than One Dashboard Shows
A multi-agent reservation workflow turns one user request into a chain whose weakest decision can determine the final result.
The airline example makes the operational problem concrete. A supervisor agent receives the request and routes work among three specialist agents.
That hierarchy can limit each agent’s responsibilities. However, it also means the final response depends on successful coordination across several reasoning and service boundaries.
Suppose a traveler asks to change a reservation. The supervisor must recognize the request, preserve relevant context, select the correct specialist, and pass suitable instructions.
The specialist must then call the appropriate tool with valid parameters. It must interpret the result correctly and return enough information for the supervisor to continue.
A technically successful call can still produce a bad customer outcome. The system might retrieve flights but ignore a date constraint supplied earlier in the conversation.
It might contact the correct reservation service with the wrong passenger identifier. It might also produce a plausible confirmation without completing the underlying action.
None of those outcomes necessarily causes high CPU usage or an obvious server error. Some may even return normal status codes and acceptable latency.
That is why agent evaluation needs traces, which record the steps taken during an interaction. A trace can connect the final answer with routing choices, model calls, tool activity, and supporting services.
OpenTelemetry has been developing generative AI conventions for describing model and agent activity through standardized telemetry. The effort shows why ordinary application fields are insufficient for agent workflows.
The airline design also exposes a second problem. Poor quality and infrastructure failure can produce similar symptoms from the user’s perspective.
A reservation tool might time out because its service is unavailable. Alternatively, the agent may never call that tool because it misunderstood the request.
The user only sees that the reservation failed. The operating team must determine which class of failure occurred before choosing a remedy.
That determination pressures several teams at once. AI engineers own prompts, evaluation criteria, and agent orchestration. Platform engineers own runtimes, permissions, telemetry, and dependent services.
Application owners still own the customer outcome. Security teams may control the identities and policies that allow one agent to reach another resource.
A conventional alert can send all those teams into the same incident channel without establishing where the failure began. That encourages manual log searches and competing theories.
AWS’s dual-layer design tries to supply a better starting point. A low quality score identifies a behavior worth examining, while infrastructure investigation tests one category of cause.
The approach is most valuable when teams preserve the relationship between evaluation results and operational traces. Without that connection, they merely receive two disconnected streams of alerts.
Trace continuity is especially important in asynchronous or distributed workflows. The original user request can cross several processes before an action completes.
Every handoff needs a stable identifier and useful attributes. Otherwise, investigators cannot reliably connect a low-scoring session with the exact infrastructure events that supported it.
This is where the architecture becomes more than a product demonstration. It defines an ownership contract for production agents.
The evaluation layer says whether the system behaved acceptably. The observability layer shows what happened. The investigative layer proposes why infrastructure behaved that way.
Humans must still decide whether the evidence is sufficient. They also decide whether the remedy belongs in a prompt, evaluator, service, policy, or data pipeline.
The Real Mechanism Is a Handoff Between Two Investigations
AWS’s design works only when quality scoring hands investigators a specific session, trace, and timeframe to examine.
AgentCore Evaluations is not simply another health check. It applies evaluators, meaning scoring rules or model-based judges, to an agent interaction.
An evaluator needs a defined target. Depending on the implementation, that target might include task completion, relevance, correctness, helpfulness, or another business-specific criterion.
Continuous evaluation can surface patterns across production traffic. Teams can inspect whether scores changed after a prompt revision, model switch, tool update, or deployment.
The online path is useful for detection. It transforms sampled production interactions into a quality signal that operators can trend over time.
The on-demand path supports diagnosis. An operator can evaluate a selected trace or session after discovering an unusual outcome.
AWS’s example also shows AI-assisted pattern analysis for low-scoring sessions. It presents prompt improvement recommendations as a possible response to the evidence.
That recommendation remains a hypothesis. A suggested prompt can address unclear instructions, but it cannot repair a broken permission or unavailable dependency.
The next handoff therefore matters. When the evidence points toward an operational fault, AWS DevOps Agent investigates the relevant cloud environment.
Its role is broader than summarizing one log line. The demonstrated interface builds a topology graph, reviews CloudWatch data, and follows the path associated with the failure.
A topology matters because modern agent applications rarely run as one isolated process. They depend on runtimes, APIs, identity controls, data stores, and network paths.
The DevOps agent then presents an identified root cause and a traced failure path. It also offers prevention or remediation steps for consideration.
That sequence mirrors an experienced incident responder’s workflow. Establish the affected path, collect correlated signals, test possible causes, and recommend a response.
Automation changes the speed and breadth of that investigation. It does not change the need to validate the conclusion against source telemetry.
The dual-layer mechanism is strongest when the first system narrows the second system’s search area. A low score without trace context leaves the infrastructure agent with too much ambiguity.
The reverse is also true. A resource anomaly without a quality signal may have no meaningful effect on users.
This creates a practical monitoring pipeline:
Instrument every agent and tool handoff with traceable identifiers.
Evaluate production interactions against explicit quality criteria.
Detect low scores, score shifts, or repeated failure patterns.
Inspect the affected session and its agent trajectory.
Escalate suspected infrastructure causes for autonomous investigation.
Review the evidence before changing production behavior.
rerun evaluations after remediation to measure the outcome.
The final step closes the lifecycle. A fix is incomplete until teams can show that it improved the targeted behavior without damaging another task.
That principle applies to prompt changes as much as infrastructure changes. A revised supervisor instruction might repair routing for cancellations but weaken routing for new reservations.
Evaluation sets should therefore include representative tasks and important edge cases. Production sampling can reveal unexpected behavior, while controlled regression suites protect known requirements.
Amazon’s broader AgentCore documentation describes a collection of services for deploying and operating agents. Evaluations belongs inside that larger operating environment.
The architecture also depends on CloudWatch as a telemetry foundation. AWS’s observability guidance covers metrics, logs, alarms, and traces used to understand applications and resources.
Those foundations explain the product split. AgentCore focuses on the behavior of the agent system, while the DevOps investigation focuses on the environment carrying that behavior.
The division is useful, but it creates an integration obligation. Teams need consistent trace context, access controls, retention policies, and escalation rules across both layers.
Without those controls, autonomous analysis can generate polished explanations that remain difficult to audit. The mechanism succeeds when every conclusion points back to observable evidence.
Evaluation Scores Can Create Their Own Blind Spots
The largest uncertainty is not whether AWS can calculate a score, but whether that score represents the outcome a business actually values.
An evaluation metric is a compressed judgment. It turns a complex interaction into a label, category, or number that teams can monitor.
That compression makes operations manageable. It can also conceal disagreement about what success means.
A reservation assistant might receive a high relevance score while violating a fare rule. It could sound helpful while failing to preserve a passenger’s selected seat.
A generic evaluator might reward a concise answer that omits an important warning. Another evaluator could penalize a correct response because its expected wording is too narrow.
Model-based judges introduce additional uncertainty. Their ratings can vary with the judging model, instructions, context, and examples used to define the rubric.
Teams therefore need to validate evaluators against human-reviewed cases. They should measure disagreement, inspect false positives, and track false negatives for high-impact tasks.
Thresholds also require context. A small decline in an average score may reflect a genuine regression, a new traffic mix, or normal judging variation.
Aggregates can hide concentrated harm. An airline assistant might perform well overall while failing disproportionately on international changes or complicated family itineraries.
Sampling creates another blind spot. Continuous evaluation is operationally useful, but teams may not score every interaction with every evaluator.
The selected sample must cover valuable, risky, and uncommon workflows. Otherwise, dashboards will favor the common requests they observe most often.
The AWS example demonstrates an architecture rather than independent proof that the combination catches every failure class. Organizations still need to test it against their own incidents.
Infrastructure investigation has similar limits. Correlation across logs and resource relationships can identify a persuasive failure path without establishing the only possible cause.
Incomplete telemetry can distort the analysis. A missing span, inconsistent timestamp, or absent application attribute can make the wrong component appear responsible.
Access design also affects visibility. An investigation agent needs enough permission to inspect relevant resources, but unrestricted access would create an unnecessary security risk.
Enterprises should grant read access narrowly and record the agent’s queries. Any automated remediation deserves stronger safeguards than an investigation.
NIST’s AI risk profile emphasizes measurement, monitoring, documentation, and human oversight for generative AI systems. Those practices remain relevant when AI evaluates or investigates another AI system.
The most defensible deployment separates diagnosis from execution. Let the system collect evidence and recommend action, then require approval for material production changes.
That boundary should reflect the possible impact. Restarting a stateless development service differs from changing an identity policy or altering a booking workflow.
Teams also need to manage sensitive data inside traces. Agent sessions may contain customer details, reservation information, tool arguments, or model outputs.
Evaluation and observability pipelines should minimize unnecessary content. Retention, encryption, access, and redaction policies must apply to the monitoring data itself.
Vendor concentration presents a strategic tradeoff. The demonstrated pattern uses AWS services across runtime, evaluation, telemetry, and infrastructure investigation.
That integration can reduce operational friction for workloads already centered on AWS. It can also make cross-cloud investigation and migration more complicated.
Open telemetry formats can reduce some coupling. Standardized trace identifiers and semantic fields make it easier to export evidence into existing observability systems.
However, a standard event schema does not standardize evaluation meaning. Teams must still define quality in terms of their application, users, and risks.
The decisive question is therefore not whether an organization has an evaluation dashboard. It is whether engineers can explain what each score measures and when it fails.
A credible program will maintain versioned evaluators, benchmark them against human judgments, and review them after application changes.
It will also preserve raw evidence long enough to audit major incidents. Scores should guide attention, not replace the underlying interaction record.
AWS DevOps Agent Pressures Existing Observability Workflows
The competitive pressure falls less on one monitoring vendor and more on the practice of separating AI quality from infrastructure operations.
Many organizations already use application performance monitoring, centralized logging, distributed tracing, and incident-management platforms. Those systems remain central to production operations.
The AWS approach does not make them obsolete. It argues that their existing signals need an agent-specific quality layer and a more autonomous investigative interface.
Cloud providers are well positioned to make that argument. They can access service relationships, native telemetry, identity context, and deployment metadata within their platforms.
Independent observability vendors hold a different advantage. They often provide one operating view across multiple clouds, services, models, and application frameworks.
The strategic contest is therefore between integrated cloud context and portable operational context. AWS’s pattern favors deep integration across its own services.
A cross-platform stack favors consistent investigation when agents use several model providers or run across different environments. Neither route removes the need for application-specific evaluation.
Agent development platforms also compete for the behavioral layer. Some provide tracing, datasets, prompt experiments, evaluators, and regression testing around agent workflows.
AWS can connect those concerns directly to its managed runtime and operational services. The value depends on how smoothly teams can follow one trace across every boundary.
That continuity will determine whether the dual-layer idea becomes a routine workflow or another set of dashboards. Operators resist tools that increase context switching during incidents.
The DevOps agent also changes expectations for incident response. A system that autonomously builds a topology and proposes a cause can shorten initial investigation.
It can also increase alert volume if teams trigger investigations from poorly calibrated evaluations. Low-quality detection will feed low-quality incidents into the second layer.
This makes evaluator governance part of operational governance. AI teams cannot tune scores independently from the engineers who receive their alerts.
Shared runbooks should define when a score creates a ticket, when it starts an investigation, and when it only updates a trend.
They should also distinguish quality incidents from infrastructure incidents. A quality regression may require a prompt rollback, model rollback, tool fix, or data correction.
An infrastructure incident may require capacity, configuration, permission, networking, or service remediation. Some incidents will span both categories.
A useful taxonomy prevents every low score from becoming a cloud outage. It also prevents every timeout from being dismissed as infrastructure noise.
The four-agent airline example is valuable because it exposes this ambiguity. The supervisor may make a poor routing decision when every service is healthy.
A specialist may make the correct decision but fail because its dependency is unhealthy. Both outcomes can look identical to the customer.
AWS’s contribution is an explicit mechanism for separating those possibilities. The market pressure follows from that operational claim.
Observability providers will need to show how their platforms evaluate agent behavior, not merely display token use and model latency.
Agent platforms will need to show how their traces connect with infrastructure evidence. A reasoning trace that ends at a tool call cannot explain what happened inside the tool.
Enterprise buyers should assess coverage across that entire chain. They need to know which layer detects the failure, which layer investigates it, and which team owns remediation.
They should also ask whether exported telemetry retains enough meaning outside the original platform. Portability affects audits, incident review, and future architecture choices.
For knowledge-heavy engineering teams, a searchable record of prior incidents, decisions, and runbooks can complement live telemetry. A technical knowledge base helps preserve the human context that monitoring tools rarely capture.
That context does not replace traces. It explains why thresholds exist, which remedies previously failed, and who approved important operational changes.
The winning workflow will connect machine evidence with institutional memory. Neither layer is sufficient alone.
Three Signals Will Show Whether the Two-Layer Model Works
The next test is whether AWS customers can turn the demonstration into measurable, auditable production improvements.
The first signal is evaluation stability across application changes. Teams should watch whether AgentCore scores remain meaningful after prompts, models, tools, and traffic patterns change.
Stable does not mean frozen. It means score movements correspond with human-reviewed improvements or regressions rather than unexplained evaluator drift.
Evidence of repeatable calibration would strengthen AWS’s case. Frequent threshold changes without clear validation would weaken confidence in the behavioral layer.
The second signal is trace continuity from a poor outcome to an infrastructure finding. The reservation example depends on carrying useful context across the supervisor, specialist agents, tools, and AWS resources.
Customers should look for investigations that begin with one low-scoring session and end with a specific, evidence-backed cause. The path should be reproducible by a human operator.
Consistent trace linkage would support the dual-layer model. Missing spans and weak correlation would leave teams performing the same manual searches behind a new interface.
The third signal is the rate of accepted remediation recommendations. AWS DevOps Agent can present a root-cause narrative and prevention steps, but operators must decide whether those recommendations are correct.
Teams should track how often engineers accept, modify, or reject proposed actions. They should also measure whether accepted changes prevent recurrence without creating new quality failures.
High acceptance alone is not enough. The better measure combines accuracy, time saved, recurrence, and post-change evaluation results.
These signals should appear in operational reviews rather than promotional demonstrations. Production evidence will reveal whether autonomous investigation reduces incident duration or simply accelerates the first hypothesis.
Organizations adopting the pattern should begin with bounded workflows. A reservation lookup presents less risk than an irreversible booking or payment action.
They can define a small set of business outcomes, create human-reviewed evaluation cases, and instrument every handoff. They can then connect low scores to read-only infrastructure investigations.
That staged approach produces evidence before granting broader authority. It also exposes missing telemetry and weak evaluation criteria while the impact remains limited.
Teams should document the relationship between each evaluator and a customer outcome. A relevance score should not stand in for transaction correctness.
They should also version prompts, evaluators, models, tools, and runbooks together. Otherwise, an incident review cannot reconstruct which operating assumptions applied at the time.
AWS agent monitoring offers a useful answer to an increasingly visible problem. An agent can be available, fast, and wrong, while infrastructure can be healthy or broken beneath the same symptom.
AgentCore Evaluations and AWS DevOps Agent divide that problem into behavioral measurement and operational investigation. The design is credible because each layer addresses a different question.
Its success will depend on the handoff between them. Can a low score lead to the right trace, the right infrastructure evidence, and a verified improvement?
That is the question enterprise teams should test next. Start with one consequential workflow, define what success means, and ask whether the two layers produce a clearer decision than your current incident process.
