Amazon Bedrock AgentCore Evaluations: Why Agent Success Needs More Than One Score
AWS published an agent evaluation playbook on November 10, arguing that completing a task and completing it safely require separate measurements. Amazon Bedrock AgentCore Evaluations brings AWS-authored, DeepEval, and AutoEval evaluators into the same service. The practical question is which behavior each evaluator measures, and whether its evidence supports the business decision attached to its score.
An evaluator is a scoring mechanism that checks an agent’s behavior against defined criteria. A successful support conversation can still contain an unauthorized disclosure. A research answer can sound convincing while omitting a requested comparison. These failures challenge the assumption that a favorable overall rating means an agent is ready for production.
The pressure falls on developers and enterprise buyers who must explain what their quality dashboards actually establish. Anthropic’s earlier evaluation guidance provides a useful reference: grading an agent involves examining its actions and resulting state, alongside its responses. AWS’s playbook addresses the same underlying problem through a managed evaluator catalog. Running the checks becomes easier; deciding what counts as acceptable behavior remains a product responsibility.
Amazon Bedrock AgentCore Evaluations puts business outcomes ahead of metric names
The November publication is an evaluation playbook, rather than the launch of AgentCore itself. Its contribution is a concrete way to connect business expectations with observable agent behavior.
AWS’s agent evaluation playbook recommends starting with an outcome, identifying the behaviors behind it, and selecting evaluators at suitable levels. That framing matters because attractive metric names can conceal a mismatch between what teams measure and what customers need.
A metric becomes useful when its result answers a specific operational question.
Consider an illustrative customer-support deployment. The business wants customers to resolve eligible refund requests without unnecessary escalation. A friendly answer is relevant, but it cannot establish whether the refund happened. Likewise, a completed refund cannot establish whether the agent disclosed another customer’s information.
Those outcomes require different evidence. The conversation shows what the customer requested and what the agent promised. Tool results show whether the requested operation succeeded. Policy requirements establish which actions were permitted. A useful assessment connects those records rather than treating fluent language as a substitute for them.
The same issue appears in research workflows. An agent might produce a readable market summary while skipping the requested source comparison. The output looks finished because it has a conclusion. The underlying task remains incomplete because the user asked for something more specific.
This creates pressure on product managers as much as engineers. If the acceptance criteria only say “helpful answer,” evaluators inherit that ambiguity. A dashboard cannot resolve a disagreement that the product specification never addressed.
A stronger specification identifies observable failures. It distinguishes a missing answer from an incorrect answer, an inappropriate action from a failed action, and a justified refusal from unnecessary abandonment. These distinctions give the evaluator stack a purpose.
This is also why maximizing the number of enabled checks is an unreliable measure of evaluation maturity. Overlapping metrics can increase review work without covering a missing requirement. A small, deliberate stack can provide clearer evidence than a large catalog selected by name.
The news value lies in that shift of responsibility. AWS supplies more evaluation machinery inside an existing platform. Teams still need to establish which results should block a release, trigger investigation, or inform a later improvement.
For buyers, the resulting question is concrete: when a vendor reports agent quality, which customer outcome does that measurement represent? Without that connection, a score describes the test more clearly than it describes the product.
The AgentCore evaluator comparison turns on control and coverage
An AgentCore evaluator comparison starts with ownership. AWS’s built-in evaluators use predefined models and prompts that customers cannot modify directly. They provide a standardized starting point for common behaviors, with less configuration work.
The managed third-party option follows a similar operating model. AgentCore hosts the evaluator logic and chooses the model. Customers do not select the managed library version. AWS explicitly distinguishes its benchmarked built-in evaluators from third-party evaluators, whose quality it does not endorse.
According to the third-party evaluator documentation, DeepEval covers behaviors including task completion, conversation completeness, knowledge retention, tool use, and personal-information leakage. AutoEval contributes specialized checks named Security, Possible, and Humor.
The meaningful choice is which failure a check detects, followed by who controls how it runs.
ConversationCompleteness addresses whether the agent handled requests across a conversation. KnowledgeRetention concerns information shared earlier. PIILeakage checks whether a response exposes personal information. These labels describe distinct concerns; they should not become interchangeable indicators of general reliability.
AutoEval’s Possible check asks whether an agent attempts a solution or declares the task impossible. That creates an important interpretive limit. An attempt can still be unsafe, unauthorized, or unsuccessful. A favorable result on that dimension cannot settle those separate questions.
Similarly, Security assesses malicious responses. Its presence does not establish that the entire application resists every attack. An application also includes permissions, tools, external services, and stored data beyond any response being scored.
For teams needing model control, custom derived evaluators retain a base evaluator’s logic while using customer-selected inference. AWS says the base owns the instructions, scale, and evaluation level. A separately authored custom evaluator provides a different route for defining a specialized rubric.
That distinction prevents a common misunderstanding. Changing the judge model and rewriting the judging criteria are separate interventions. Each can change results, but they answer different requirements and create different maintenance obligations.
Imagine a procurement assistant that follows an organization’s approval policy. A standard tool-use check might assess whether its selected tool fits the request. A business-specific check might assess whether the purchase required approval before execution. Neither result substitutes for the other.
This AgentCore evaluator comparison also exposes a continuity issue for existing DeepEval users. Familiar metric names help connect development and production practices. They do not, by themselves, establish identical scores across local and managed environments.
The judge model, library version, available context, and configuration can affect interpretation. Teams comparing environments should treat agreement as something to measure. A shared name is useful organizational shorthand, but it is insufficient evidence that an existing threshold transfers unchanged.
The evidence level determines which failures become visible
An agent trace is a record of execution containing model interactions and actions. A session groups the conversation across turns. A tool call records a particular operation and its arguments. These levels offer different views of the same workflow.
AWS’s prompt definitions distinguish session, trace, and tool-level inputs. Its evaluation prompt templates show that session checks receive conversation-wide evidence, while other checks target particular responses or calls with relevant context.
Selecting the wrong evidence level can make a valid score answer the wrong question.
A relevant response can satisfy the latest message while abandoning an earlier request. Conversely, a conversation can eventually reach the correct outcome after an intermediate tool mistake. A session result and an action-level result describe different aspects of that interaction.
Consider an illustrative travel assistant asked to find an itinerary and respect a traveler’s accessibility requirements. A later response might answer a baggage question correctly. That response alone cannot establish whether the proposed itinerary still satisfies the original constraints.
The distinction is especially important when teams aggregate results. Averaging response quality across a conversation can obscure whether the overall goal was achieved. A conversation with consistently polite answers can still end without a usable booking.
Tool selection and tool arguments also need separate attention. Choosing an order-lookup tool is different from supplying the correct account identifier. A reasonable tool choice with incorrect arguments can produce an operational failure despite sound conversational behavior.
Skills add another evidence requirement. A skill is a reusable instruction file loaded by an agent for a task. AWS’s skill checks evaluate selection and adherence at the tool-call level. Instruction-following assessment requires the loaded skill body and the subsequent session context.
A missing result therefore deserves investigation before celebration. If the necessary invocation or instruction content never reaches evaluation, silence cannot establish compliance. Teams need to distinguish an assessed success from an unassessed action.
Trajectory checks introduce another perspective. A trajectory is the sequence of tool actions taken during execution. AWS describes exact-order, in-order, and any-order comparisons against an expected path. These checks require a reference trajectory, rather than merely a plausible final response.
The business must decide which sequence constraints are essential. Requiring approval before a purchase expresses a meaningful dependency. Requiring an arbitrary ordering of independent lookups can penalize acceptable alternatives.
This is where the playbook’s outcome-first approach becomes consequential. An expected path should encode business requirements, not simply reproduce whichever route a developer happened to record first.
The same reasoning applies to knowledge work. Teams building a searchable knowledge base need to distinguish document retrieval from a complete, supported answer. Finding a relevant file is an intermediate action. Whether the agent used it appropriately requires additional evidence and a different judgment.
AgentCore evaluation modes separate release evidence from live monitoring
AgentCore evaluation modes determine when evidence becomes available and how teams use it. On-demand evaluation accepts session spans supplied by the caller. Online evaluation monitors sampled production activity. Batch evaluation processes multiple sessions through service-side orchestration.
AWS’s batch evaluation documentation identifies an important boundary: on-demand and batch evaluation can accept ground truth, while online evaluation does not. Ground truth means supplied reference material, such as an expected response or success assertions.
A production monitoring score and a reference-based release test answer different questions.
A fixed regression suite, meaning tests for previously supported behavior, lets teams compare changes under controlled conditions. The same cases can expose whether a prompt revision improves one requirement while damaging another.
Live monitoring reveals something else: what happens when actual users bring requests the test authors did not anticipate. Its value comes from contact with production behavior, including unfamiliar language, changing workflows, and unexpected combinations of requirements.
Neither replaces the other. A curated dataset can become disconnected from current usage. Production samples can miss uncommon failures or lack the reference information needed for a definitive judgment.
For an illustrative support agent, a release dataset might contain known refund eligibility scenarios. Each case includes the relevant policy and expected outcome. Online monitoring then examines how the deployed agent handles real conversations, where no prepared answer accompanies every request.
A deterioration in live scores should prompt investigation. It does not automatically establish whether the model changed, user requests became harder, or captured evidence became incomplete. Interpreting the signal requires checking those alternatives.
Batch results also need scrutiny below the average. AWS provides aggregate summaries and session-level detail. An aggregate improvement can coexist with a serious failure in a particular workflow, language pattern, or request category.
Suppose an agent handles routine requests better after a model change but becomes less reliable when users revise earlier instructions. The overall average might improve. Whether that release is acceptable depends on the importance and frequency of the affected behavior.
Multi-dimension custom evaluators address a related reporting problem. They return separate ratings for named criteria from a shared judge call. AWS documents that these dimensions share a model and evaluation level.
The advantage is clearer visibility into independently failing criteria. Resolution, policy compliance, and tone can remain distinct rather than disappearing inside a blended rating. Teams can then attach different consequences to each result.
The limitation is shared execution. A failed judge call can affect every dimension, and criteria requiring different evidence or models belong elsewhere. Consolidating checks therefore creates a design tradeoff, even when it reduces repeated reading of the same context.
For engineering leaders, the implication is organizational. Release gates, production alerts, and exploratory analysis should have explicit roles. Calling all of them “evaluation” does not make their evidence equally suitable for every decision.
Managed scoring leaves judgment, privacy, and operating costs with the team
AgentCore needs conversation content to reconstruct and assess behavior. AWS’s telemetry documentation identifies prompts, completions, and tool inputs and outputs as relevant evidence. Telemetry means the execution records an application exports for monitoring and analysis.
That creates a practical tension. The information needed to detect a disclosure can itself contain sensitive material. Teams evaluating agents that handle private records need to consider how evaluation evidence is collected, accessed, retained, and reviewed.
A managed evaluator reduces infrastructure work without removing responsibility for the evidence or the verdict.
Redaction illustrates the difficulty. Removing sensitive information can reduce exposure, but it can also remove the context needed to assess whether disclosure occurred. The appropriate design depends on which question the evaluator must answer.
For example, a masked account identifier might preserve enough structure to check whether the agent used the requested account consistently. It might be insufficient to determine whether the response exposed an unauthorized person’s details. That difference should shape the evidence design before rollout.
Model-based judging adds another limit. An LLM-as-a-judge uses a language model to assess behavior against instructions. Its result is a model judgment, rather than an automatic guarantee of factual or operational correctness.
Anthropic’s agent evaluation guidance describes model graders as flexible but nondeterministic, and emphasizes calibration with human reviewers. It also distinguishes response transcripts from the resulting state of the environment. This provides a useful counterweight to any interpretation that managed scoring eliminates verification work.
The comparison is about emphasis rather than a measured contest between providers. AWS’s playbook organizes evaluator selection and execution. Anthropic’s guidance stresses evaluation design, grader tradeoffs, and checking outcomes. Neither source establishes that one managed score universally predicts agent reliability.
Human calibration is particularly important when reviewers disagree about success. A judge cannot reliably operationalize a policy whose owners interpret it differently. Reviewing disagreements can reveal unclear instructions, incomplete evidence, or a rubric that rewards the wrong behavior.
Operational cost deserves the same specificity. More evaluators, longer context, repeated calls, and larger samples can expand the work required for assessment. The relevant business question is whether each added check changes a decision or detects a meaningful failure.
A specialized evaluator that identifies a costly omission can justify its operating burden. A redundant score that moves with an existing metric may add little. Teams need evidence for that distinction rather than assuming broader coverage from a larger count.
Derived evaluators introduce additional ownership. Choosing the judge model gives teams control, but also makes model suitability their responsibility. A model that performs well as an assistant should still be assessed for the particular judging task.
Security judgments require similar caution. A clean sample does not establish universal protection, and a suspicious response deserves investigation beyond its score. Permissions and deterministic controls, meaning rules enforced without model judgment, remain important for consequential actions.
The credible promise of Amazon Bedrock AgentCore Evaluations is more accessible measurement. The available sources do not establish independent accuracy comparisons across every evaluator, workload, and failure category. Buyers should keep that verification gap visible when reviewing adoption claims.
What to watch next: agreement, coverage, and decisions
The next phase should be judged by operational evidence rather than catalog size. Over the coming months, the most useful signals concern whether evaluator results track human judgment, cover real failures, and influence release decisions.
The strongest evidence will show which mistakes evaluation caught and what teams changed because of it.
The first signal is agreement with qualified reviewers on representative cases. Published comparisons should explain the task, the evidence supplied, the judging criteria, and how disagreements were resolved. Agreement on routine examples is less informative if consequential edge cases remain unexamined.
A support deployment, for example, should separate missed disclosures from incorrect disclosure flags. These errors carry different operational consequences. Reporting only overall agreement can conceal the distinction that matters most to the business.
Consistent agreement across relevant cases would strengthen confidence in the selected evaluator stack. Persistent disagreements would point toward narrower scope, revised criteria, better evidence, or a different judge. Either outcome provides more useful information than an unexplained quality score.
The second signal is whether production failures become reproducible evaluation cases. A monitoring system earns its place when an observed omission or tool error can inform a test that protects future releases.
This requires more than storing the conversation. Teams need enough context to understand the expected behavior and reproduce the relevant conditions. Otherwise, the incident becomes an anecdote that remains difficult to use.
Evidence that organizations repeatedly convert live failures into release checks would support the playbook’s practical value. Recurring incidents that never alter the test suite would weaken the claim that continuous scoring improves reliability.
The third signal is whether evaluation changes a concrete deployment decision. A useful report would identify a regression, explain the affected behavior, and describe a resulting delay, rollback, or targeted fix.
That would connect measurement with accountability. It would also reveal whether teams preserve separate requirements when a release produces mixed results. Better tone should not automatically compensate for poorer privacy behavior.
For enterprise buyers, these signals suggest a sharper procurement conversation. Ask vendors to show a representative failed case, explain why the evaluator rejected it, and identify the product change that followed. Ask which important behaviors remain outside the current assessment.
For developers, the immediate action is to examine a recent failure your dashboard missed. Identify the evidence needed to recognize it, then decide whether an existing evaluator fits. If the failure requires a supplied reference, preserve it in a controlled test. If it depends on live context, establish how that context reaches evaluation. Use Amazon Bedrock AgentCore Evaluations to make those decisions inspectable, then review the disagreements with people responsible for the outcome. Which unacceptable behavior would still pass your current release checks?



