Amazon Bedrock AgentCore Skill Evaluation Exposes What Fluent Agents Hide

Amazon introduced Amazon Bedrock AgentCore skill evaluation on September 22, adding three checks that examine agent behavior beyond its polished final answer. The release targets a persistent testing blind spot. An agent can sound correct after choosing the wrong skill, skipping required steps, or improvising around a business procedure.

The new evaluators separate two questions that teams often collapse into one score. Did the agent select an appropriate skill, and did it follow that skill after loading it? Strands Evals adds a third, deterministic check for teams that already know which named skill a test should invoke.

That distinction puts pressure on evaluation systems centered on response quality alone. Helpfulness, relevance, and correctness still matter, but they cannot reveal every routing or execution failure. The real contest is now final-answer scoring versus trajectory-level evidence about how an agent reached that answer.

Amazon Bedrock AgentCore Skill Evaluation Splits One Failure Into Three

AWS is turning skill use into a measurable sequence, rather than treating the final response as sufficient evidence of success.

A skill is a reusable instruction package that teaches an agent a specialized procedure. It commonly includes a SKILL.md file containing its purpose, activation guidance, and required steps. A harness presents available skills, while the agent decides which one to load for a request.

This structure lets developers move detailed procedures out of a growing system prompt. A company might create separate skills for invoice reconciliation, contract redaction, incident escalation, or pull-request review. The agent loads relevant instructions when needed instead of carrying every procedure through every interaction.

Portability is part of the appeal. The open Agent Skills format gives compatible agent environments a shared way to package specialized instructions. A skill can therefore serve as an operational artifact, not merely a prompt fragment tied to one model call.

However, modular instructions introduce a chain of decisions. The agent must recognize the user’s intent, find a suitable skill, invoke it, read its contents, and complete its prescribed steps. A good final paragraph does not prove that this chain worked.

AWS and the Strands team now divide that chain across three evaluators, according to the September 22 skill evaluation release.

Skill Selection Accuracy asks whether each invoked skill was appropriate for the task. It returns a binary result for every invoked skill. This makes routing mistakes visible when an agent loads instructions intended for another workflow.

Skill Instruction Following examines how completely the agent performed an invoked skill’s prescribed steps. Its five ratings are Fully Followed, Mostly Followed, Partially Followed, Minimally Followed, and Not Followed. Their documented numeric values run from 1.0 to 0.0 in quarter-point increments.

Skill Invoked provides a narrower, deterministic assertion within Strands Evals. It checks whether the agent successfully loaded a named skill. Unlike the other two evaluators, it does not ask a model to judge appropriateness or adherence.

These measures answer different questions. A required payroll skill might never load, producing a routing failure. It might load for an unrelated travel request, producing a selection failure. It might load correctly but omit an approval step, producing an instruction-following failure.

That separation is the central change. Teams no longer have to interpret every weak outcome as a vague agent-quality problem. They can associate each pattern with a different component and a more focused repair.

A missing invocation points toward discovery rules, descriptions, or routing logic. An inappropriate invocation suggests overlapping skill scopes. A correctly selected skill with low adherence directs attention toward its steps, structure, available tools, or the underlying model.

The release does not replace existing quality evaluation. It adds another layer designed for agents whose behavior depends on dynamically loaded procedures. Output accuracy remains essential, but it becomes one part of a larger test record.

Fluent Answers Are No Longer Enough Evidence

The strongest argument for trajectory evaluation is simple: different internal failures can produce equally convincing prose.

Consider an employee asking an agent to redact a contract before external sharing. The agent might remove obvious names and return a clean-looking document. Yet the approved skill could also require checking metadata, hidden comments, tracked changes, and attachment references.

A reviewer who sees only the final document may miss those skipped checks. The response can look competent while violating the organization’s actual handling procedure. Skill Instruction Following is designed to compare the recorded behavior with each prescribed step.

The same problem appears in financial operations. An invoice-reconciliation agent might produce the correct total after informal reasoning. If the skill requires validating vendor identity and purchase authorization, the result remains procedurally incomplete.

Compliance makes this distinction especially important. Organizations rarely care only about whether one answer happened to be acceptable. They also need evidence that repeatable controls were applied in the required order and context.

Traditional software tests offer exact expectations for deterministic functions. Agents behave differently because the same prompt can lead to varied language, tool calls, and reasoning paths. AWS previously argued that one passing run shows what can happen, not what usually happens.

That variability makes aggregated response scores tempting. A team can average correctness or helpfulness across a dataset and track whether the number rises. Yet an average hides where a workflow failed and whether the same step keeps disappearing.

Per-skill results offer a more useful diagnostic unit. When an agent invokes several skills during one session, the evaluators return results for each invocation. A weak aggregate can therefore be traced to the specific skill that lowered it.

The approach also changes how teams write skills. A vague paragraph may be understandable to a human author but difficult to evaluate consistently. Numbered, observable steps give the judge clearer evidence and make omissions easier to identify.

This does not mean every internal thought becomes available. The evaluation relies on recorded trajectories and traces, including visible messages, skill-loading actions, and tool calls. Private model reasoning is neither required nor exposed.

The relevant evidence is operational. Did the agent load the skill? Which skill did it choose? Do recorded actions show that it completed the prescribed checks? That evidence is more actionable than speculation about hidden reasoning.

This shift resembles the difference between checking a finished calculation and auditing the controls around it. Both views matter, but they answer separate questions. One measures the artifact, while the other measures the process that produced it.

For teams building internal agents, the process often carries the greater organizational risk. A fluent answer can satisfy a user once. A skipped approval, disclosure, or validation step can undermine the workflow every time the same conditions recur.

The new evaluators make that procedural gap easier to name. They also create pressure for other agent platforms to expose compatible trajectories. Without observable skill events, a team cannot confidently distinguish a missing invocation from a failed extraction.

Strands Evals Brings the Checks Into Development

Strands Evals gives developers a local testing layer for skill routing and execution before production traffic becomes the test suite.

Strands Evals is an open source framework for evaluating agents and language-model applications. Its published capabilities include output scoring, trajectory analysis, tool assessment, simulations, experiments, and trace-based evaluation.

The project’s evaluation repository now documents all three skill checks. Developers can run Skill Selection Accuracy and Skill Instruction Following against a recorded session or raw message trajectory.

The judge-based evaluators read the trajectory rather than rerunning the agent. This supports investigation after a failure and comparison across saved sessions. It also separates expensive agent execution from repeated analysis of the same record.

Skill Invoked serves a different testing need. If a regression case has one known routing requirement, developers can assert that the expected skill loaded. The check is deterministic and does not require a judge model.

That makes it suitable for a release gate. A customer-support request involving account closure should consistently load the approved closure skill. If a revised description prevents invocation, the regression test can fail before deployment.

Selection accuracy remains useful when more than one skill might reasonably apply. It asks whether an invoked skill fits the task instead of comparing only against one fixed name. That flexibility accommodates catalogs with related procedures and legitimate routing variation.

Instruction following then tests the next stage. The evaluator identifies prescribed steps in the loaded skill and labels each one covered, partial, or skipped. It uses those judgments to produce the five-level overall rating.

The combination creates a compact testing matrix.

A high selection score with weak instruction following means routing worked, but execution did not. The agent found the right procedure and then skipped or only partly completed its requirements.

Weak selection with strong instruction following means the agent followed the loaded procedure, but that procedure was wrong for the request. Improving the skill’s internal wording would not solve that routing error.

A missing invocation requires special handling. AWS notes that the two judge-based evaluators do not return a score when no skill was invoked. Teams should pair them with Skill Invoked when a named skill is mandatory.

That behavior prevents a misleading success. An evaluator cannot judge adherence to instructions that were never loaded. However, an empty result can disappear inside a dashboard unless the test suite explicitly treats non-invocation as a failure.

Strands also places an instrumentation burden on the harness. Its extractor must recognize available and selected skills from the trajectory. The project supports several known environments plus the generic pattern of reading a SKILL.md file.

Developers should verify extraction before trusting a score. A harness with unrecognized skill signals can produce empty results even when the agent used a skill. That is an observability gap, not evidence of correct behavior.

This caveat matters for teams integrating custom orchestration layers. Evaluation quality depends on faithful recording of events. A missing trace attribute can resemble a missing agent action unless teams validate the telemetry contract first.

The development workflow therefore has two stages. First, confirm that the evaluator can see the catalog, invocation, skill content, and subsequent actions. Second, measure whether those actions fit the task and satisfy the instructions.

For engineering teams maintaining local technical workflows, the change also reinforces the value of a searchable engineering knowledge base. Skills can encode procedures, while maintained source material supplies the facts those procedures operate on.

AgentCore Moves Skill Evaluation Into Production Traces

AgentCore extends the same routing and adherence questions from curated tests to staged sessions and sampled live traffic.

Amazon Bedrock AgentCore Evaluations is a managed service for assessing agent behavior across development and production. It consumes OpenTelemetry traces, which record structured events such as model calls, tool use, and agent operations.

OpenTelemetry matters because it reduces dependence on one agent framework. AgentCore documentation says the service supports integrations including Strands and LangGraph through OpenTelemetry and OpenInference instrumentation.

This architecture gives the release a broader role than a Strands-only feature. Strands Evals handles test cases and recorded development trajectories. AgentCore can evaluate compatible traces from deployed agents, including sessions produced outside the Strands framework.

AWS offers three evaluation modes. On-demand evaluation investigates selected sessions or validates a recent change. Batch evaluation processes multiple stored sessions to establish a baseline or compare a catalog revision.

Online evaluation continuously samples production traffic. Teams choose evaluators, a data source, filters, and a sampling rate. AgentCore then applies those evaluations as matching traces arrive.

The evaluation modes support different operational questions. A developer can inspect one failed session, score a stored population, or monitor behavior that only appears among real users.

That progression addresses a common gap in agent testing. Curated prompts reflect what designers expect people to ask. Production requests contain abbreviations, missing context, unusual phrasing, and combinations that a test author did not anticipate.

Skill catalogs also change over time. A new skill can overlap with an older description, shifting routing even when neither skill’s internal steps changed. AWS describes this as catalog drift.

Online evaluation can reveal that drift through declining selection scores. Teams can then inspect which skill started attracting unsuitable requests. The repair might involve narrowing one description or clarifying boundaries between neighboring skills.

Long sessions create another concern. An agent might follow a skill reliably near the start of a conversation but lose track of steps as context accumulates. Production traces expose those conditions more naturally than isolated test prompts.

The managed service also supports targeted sampling. AWS documentation says teams can evaluate a percentage of sessions or apply conditional filters. That lets operators focus on sensitive workflows without processing every interaction.

However, sampling changes the meaning of the dashboard. A low-volume or narrowly filtered evaluation might miss rare failures. Teams need to record what traffic qualified and avoid presenting a sampled score as complete coverage.

The production path also depends on correct telemetry. AgentCore organizes interactions into sessions, traces, and spans. A session contains a conversation, a trace covers an exchange, and spans represent individual operations.

Skill evaluation needs enough information to reconstruct what was available, what loaded, and what happened afterward. If instrumentation omits the skill content or invocation signal, the judge lacks the evidence required for a defensible result.

AWS’s AgentCore guidance describes a unified trace format scored with model-based evaluators. That standardization simplifies operations, but it cannot recover events the application never recorded.

Security teams will also need to examine trace contents. Skill text can contain internal procedures, and conversation records can include sensitive user data. Evaluation expands the value of telemetry while raising the stakes of access controls and retention choices.

The result is a lifecycle model rather than a single test. Developers can establish deterministic gates locally, compare stored sessions before release, and watch sampled behavior after deployment. Each layer catches a different class of failure.

The New Scores Still Need Their Own Evaluation

Model-based judges add diagnostic detail, but they do not turn procedural compliance into an objective fact.

Skill Selection Accuracy and Skill Instruction Following rely on a judge model. The judge reads the task, available evidence, and skill instructions before producing a rating. Its output remains an interpretation of the recorded trajectory.

That interpretation can vary with ambiguous steps. A skill might say, “verify the customer’s status before proceeding,” without defining acceptable verification evidence. One judge may treat a database lookup as sufficient, while another expects explicit confirmation.

The five-level adherence scale provides nuance, but it can also create false precision. A 0.75 rating looks exact even when the underlying distinction between Mostly Followed and Partially Followed depends on judgment.

Teams should therefore calibrate the evaluator against human-reviewed examples. The goal is not perfect agreement on every edge case. It is a stable rubric that reflects the organization’s actual procedural priorities.

Skills should make important steps observable. “Consider relevant policy” is difficult to verify. “Retrieve the current policy, compare the request with three eligibility conditions, and record the result” creates clearer evidence.

Negative cases matter as much as positive ones. A selection benchmark should include requests that resemble a skill’s domain but should not invoke it. Otherwise, a broad description can score well by activating on every nearby task.

Catalog-level testing is also essential. Evaluating one skill in isolation says little about routing when ten similar choices appear together. The relevant test environment must resemble the catalog agents will actually see.

The deterministic Skill Invoked check has its own limitation. It proves that a named skill loaded, not that loading was appropriate or useful. A team can achieve perfect invocation while still selecting the skill for the wrong requests.

Likewise, strong instruction following does not guarantee a correct answer. A flawed skill may prescribe the wrong steps. The agent can execute those steps faithfully and still produce an unsafe or inaccurate result.

That is why response-level evaluation must remain beside skill evaluation. Teams still need correctness, faithfulness, harmfulness, tool-parameter checks, and domain-specific validation. Procedure adherence is one dimension of reliability.

The official prompt templates make the scoring logic inspectable. They show that the adherence judge identifies steps, labels supporting evidence, and maps the result onto five ratings.

Transparency helps teams understand the evaluator, but it does not replace validation. Organizations should compare judge results with expert review before using scores for sensitive release decisions.

Cost and latency also shape production use. Judge-based evaluation requires additional model processing after the original agent run. Sampling and filters can control that load, but they also reduce coverage.

Teams should avoid collapsing every evaluator into one headline score. A single composite number recreates the ambiguity this release is designed to remove. Selection, invocation, adherence, and output quality should remain visible as separate signals.

The release also leaves governance questions outside its scope. It does not decide who may author a skill, approve a revision, or define a required procedure. Evaluation can reveal deviation only after an organization establishes an authoritative baseline.

A mature workflow will version skills alongside tests and rubric changes. Otherwise, teams cannot tell whether a score moved because the agent changed, the instructions changed, or the evaluator changed.

Amazon presents the checks as diagnostic tools, not independent proof of compliance. That is the right boundary. They make agent behavior more reviewable, while accountability still rests with the people defining and validating the workflow.

Three Signals Will Show Whether Skill Evaluation Works

The next test is whether teams can turn per-skill evidence into safer releases, faster diagnosis, and better skill catalogs.

The first signal is adoption of deterministic routing gates in development. Teams should identify workflows where one named skill is mandatory and add Skill Invoked assertions to regression suites.

If those gates catch catalog changes before deployment, the case for skill-aware testing becomes stronger. If extraction problems produce frequent empty results, instrumentation will remain the immediate obstacle.

The second signal is whether production selection scores reveal catalog drift. New skills often arrive with broad descriptions because authors want them to trigger reliably. Those descriptions can steal requests from existing procedures.

A useful production system should show which invocations became inappropriate after a catalog update. Teams should then be able to connect the decline to a specific description, overlap, or request pattern.

Evidence of repeatable diagnosis would strengthen AWS’s central claim. Dashboards that only show a lower aggregate without identifying the affected skill would weaken it.

The third signal is agreement between Skill Instruction Following and expert review. Organizations need to compare the judge’s step-level labels with judgments from people who understand the procedure.

Consistent agreement would justify wider use in release gates and online monitoring. Frequent disagreement would suggest that skill steps, trace evidence, or the evaluator rubric need further work.

Teams should begin with a small catalog and a deliberately varied test set. Include clear matches, near misses, requests requiring no skill, and multi-skill workflows. Run each scenario more than once because agent behavior remains non-deterministic.

Record four outcomes separately: whether the expected skill loaded, whether every invocation was appropriate, whether required steps were followed, and whether the final result was correct. This structure preserves the diagnostic value of the new evaluators.

Then inspect disagreements instead of averaging them away. A correct answer with skipped steps may expose a latent operational risk. A poor answer after faithful execution may reveal a defective skill rather than a weak model.

Production monitoring should start with sensitive or high-volume workflows. Use filters and sampling deliberately, and document what the score population excludes. Keep expert review available for severe failures and disputed ratings.

Amazon Bedrock AgentCore skill evaluation matters because it changes what counts as evidence. Fluent output is still valuable, but it no longer settles whether an agent followed the organization’s procedure.

The practical question is now yours: can your team explain which skill an agent selected, why that choice fit, and which required steps the trace proves it completed? If not, build that evidence into the next test cycle before adding more skills.