Amazon Bedrock AgentCore Evaluation Turns Agent Regressions Into Failed Pull Requests

Amazon has moved Amazon Bedrock AgentCore evaluation into a working GitHub Actions quality gate, replacing subjective agent checks with four scored tests.

The reference pipeline deploys an AI agent and an OAuth-protected Model Context Protocol server, then invokes the agent with representative prompts. It scores the resulting traces and fails the pull request when any required metric falls below a configured threshold.

That changes the argument around agent testing. The immediate contest is not AWS against another cloud provider. It is automated, repeatable evaluation against the manual spot checks that many development teams still use before merging agent changes.

Amazon published the implementation on September 8, 2026. The evaluation pipeline connects AgentCore Runtime, AgentCore Evaluations, Amazon Cognito, CloudWatch, AWS CDK, and GitHub Actions.

The important result is not another testing dashboard. A failed response can now become a failed software build, before changed agent behavior reaches a shared environment or production.

Amazon Bedrock AgentCore Evaluation Becomes a Merge Gate

AWS has turned agent quality from a review-time opinion into a pull-request condition.

The reference workflow runs when a pull request targeting the main branch changes agent code, MCP server code, infrastructure, or evaluation scripts. It creates a temporary development stack containing two AgentCore runtimes.

One runtime hosts a Strands-based agent. The other hosts an MCP server, which exposes tools through a standard protocol for connecting AI applications with external capabilities.

A shared Amazon Cognito user pool protects both runtimes. The workflow deploys this infrastructure through AWS Cloud Development Kit, then reads runtime identifiers and authentication endpoints from the deployment outputs.

GitHub Actions obtains temporary AWS credentials through OpenID Connect federation. OIDC lets GitHub exchange a short-lived identity token for an AWS role, avoiding a permanent AWS access key in repository secrets.

The workflow still stores the AWS role ARN as a GitHub secret. However, the resulting AWS credentials are temporary and constrained by the role’s trust and permission policies. GitHub documents this OIDC security model.

After deployment, the pipeline waits for both runtimes to become ready. This pause is operationally significant because a newly created runtime cannot immediately accept requests.

AWS says an early invocation returns a 424 Failed Dependency response. The supplied workflow polls the AgentCore control service and warms the MCP server before beginning its test run.

The pipeline then retrieves an OAuth access token and sends test prompts to the agent’s HTTPS endpoint. Each prompt receives a distinct session identifier, allowing the evaluation process to associate emitted traces with the correct interaction.

The sample dataset spans several kinds of behavior. It asks the agent to calculate a simple sum, return the current UTC time, find an Apple stock price, and retrieve an employee count.

Those examples probe more than answer wording. They test built-in capabilities, public MCP tools, protected business tools, tool selection, and the parameters passed into each selected tool.

The evaluation script applies four built-in evaluators:

• GoalSuccessRate asks whether the session fulfilled the user’s objective.

• Correctness compares the response with available expectations or context.

• ToolSelectionAccuracy checks whether the agent chose an appropriate tool.

• ToolParameterAccuracy checks whether the agent supplied suitable arguments.

The reference configuration uses 0.8 on a zero-to-one scale as its acceptance threshold. If a required score falls below that mark, the script exits unsuccessfully and GitHub marks the job as failed.

The workflow can also post evaluation results to the pull request. Reviewers see the measured failure alongside the code change, instead of reconstructing agent behavior from logs or local conversations.

Finally, the cleanup step runs even when evaluation fails. It destroys the temporary CDK stack so failed pull requests do not leave development runtimes running indefinitely.

This sequence creates the central tension. Manual testing can reveal an obvious problem, but it rarely provides a reproducible rule that every relevant pull request must satisfy.

Why Manual Agent Testing Is Now Under Pressure

The pipeline pressures teams that treat an agent response as something to inspect, rather than an artifact that software delivery must validate.

Traditional application tests have clear outputs. A function returns the expected value, an API matches a schema, or a database operation preserves an invariant.

AI agents complicate that model. Their responses can vary, and a correct final sentence does not prove the agent followed an acceptable route.

A tool-using agent might select the wrong service, disclose an unauthorized result, or send malformed parameters. It might also reach a plausible answer through an expensive or unsafe sequence of calls.

Prompt changes create another problem. A small adjustment to system instructions can alter tool choice, refusal behavior, response detail, or task completion across unrelated scenarios.

Model updates and dependency changes introduce similar risks. Conventional unit tests can confirm that the application runs, while missing a meaningful decline in behavior.

Teams often compensate with manual prompt sets. A developer enters several familiar questions, reads the answers, and merges when nothing looks obviously wrong.

That method has weak coverage and inconsistent judgment. It also produces little evidence about which behavior changed between two revisions.

The AWS pipeline adds a common evaluation path to every relevant pull request. It deploys the proposed code, exercises that code, collects telemetry, and applies the same evaluators.

AgentCore works with OpenTelemetry spans, which are structured records of model calls, tool activity, and other operations within an interaction. AgentCore Observability sends those traces to CloudWatch.

For on-demand evaluation, the pipeline selects spans from a particular session and passes them to the evaluation service. The evaluation modes also include online monitoring and asynchronous batch analysis.

This distinction matters for delivery teams. On-demand evaluation fits a pull request because it targets a small, controlled set of interactions and returns detailed scores.

Online evaluation serves a different purpose. It samples deployed traffic and tracks behavior after release, when real users produce requests that a test dataset did not anticipate.

Batch evaluation processes larger collections of sessions. It is better suited to baseline comparisons, periodic audits, and pre-change versus post-change analysis.

The pull-request gate does not replace those modes. It moves the earliest measurable checkpoint closer to the code change.

That approach also changes who must react to a regression. Without a gate, quality teams or users often find the problem after deployment.

With a required GitHub check, the author must address the failed behavior before merging. The feedback arrives while the relevant code and prompt decisions remain fresh.

This is especially useful for teams maintaining shared agents. A response change that looks acceptable for one developer may break a tool workflow owned by another group.

A curated evaluation dataset can preserve those expectations. It becomes an executable record of tasks the agent must continue performing.

Developers still need access to the supporting context behind those expectations. An engineering knowledge base can help teams connect evaluation cases with specifications, incidents, and design decisions.

The larger shift is organizational. Agent quality becomes part of the definition of a mergeable change, rather than an optional review conducted when someone has enough time.

OAuth Makes End-to-End Testing Harder Than Scoring

The difficult part is not requesting an evaluation score. It is reproducing a protected agent’s complete tool path inside a headless CI runner.

The reference architecture places both the agent and MCP server behind Cognito. Interactive users authenticate through an authorization-code flow and receive tokens containing scopes and custom role claims.

Those roles determine which MCP tools a user can access. A finance user and an HR user should not automatically receive the same business capabilities.

GitHub Actions has no interactive user. It cannot open a consent screen, complete a browser login, and carry a person’s role context through the test.

AWS presents three ways to handle that mismatch. Each validates a different part of the system.

The first approach evaluates stored traces. A staging process invokes the agent beforehand, captures representative telemetry, and saves those traces as test fixtures.

Pull-request jobs evaluate the fixtures without calling a live agent or MCP server. This approach is predictable and avoids the CI authentication problem.

It also has a serious limitation. The scores describe previously captured behavior, not necessarily the agent code in the current pull request.

Stored traces can validate evaluator changes or preserve a historical baseline. They cannot prove that newly changed runtime code still produces the expected trajectory.

The second approach uses a dedicated service account. An operator completes interactive authorization once and stores its refresh token in a secrets manager.

CI can then act as a known user with defined roles. That makes it possible to test authorization boundaries and tool access for a specific identity.

Refresh tokens expire or become invalid. Teams need rotation, reauthorization, and careful control over the account’s privileges.

The AWS implementation chooses a third route, machine-to-machine authentication. Cognito issues a token through the OAuth client_credentials grant.

The evaluation script sends the client ID, client secret, and requested scope to Cognito’s token endpoint. It then invokes AgentCore Runtime over HTTPS with the returned bearer token.

The MCP middleware distinguishes these tokens from user tokens. Machine tokens have scopes but no custom role claims, so the sample allows them to reach every tool.

Interactive tokens still carry roles, and the MCP server enforces tool-level restrictions for those users. The same Cognito pool therefore supports both CI access and human authorization.

This design enables live, end-to-end testing of code in the pull request. The agent can call the deployed MCP server and produce new traces for evaluation.

However, it does not test role enforcement. The CI identity bypasses those role checks by design.

That boundary is the most important caveat in the architecture. A passing pipeline shows that the machine identity can complete the evaluated tasks. It does not show that FinanceUser and HRUser receive different authorized results.

Teams that need both forms of assurance must add separate authorization tests. They can use service accounts with specific roles or direct tests against the MCP middleware.

The machine credential also becomes a sensitive asset. AWS says only the CI pipeline and agent runtime should be able to obtain it.

That claim depends on implementation details. Repository permissions, workflow approvals, IAM policies, secret exposure, fork behavior, and log redaction all influence the actual security boundary.

GitHub’s OIDC connection protects the AWS side from long-lived repository credentials. It does not eliminate every secret because the OAuth client still uses a client secret.

The IAM role should also follow least privilege. Broad deployment access to Cognito, ECR, CDK resources, Bedrock, and AgentCore can create more authority than a routine pull-request job needs.

Environment protections can narrow the risk. Teams can restrict which branches and workflows assume the role, require approval for untrusted changes, and separate deployment roles from evaluation identities.

The architecture therefore tests agent behavior and CI integration together. Authorization correctness remains a separate acceptance target.

The Quality Gate Scores Traces, Not Just Answers

Amazon Bedrock AgentCore evaluation matters because it can judge the agent’s route through a task, not only its final text.

The evaluation script uses the AgentCore starter toolkit to gather the relevant CloudWatch traces. It then applies evaluators to the selected session.

The raw Evaluate API accepts sessionSpans, a collection of telemetry spans belonging to one session. AWS warns that mixing spans from multiple sessions produces a validation error.

This session boundary lets the evaluator reconstruct an interaction. It can inspect user input, model output, selected tools, tool parameters, and the resulting response.

GoalSuccessRate operates at the session level. It asks whether the full interaction achieved what the user requested.

Correctness examines the response at the trace level. It can use an expected response when the test provides one.

ToolSelectionAccuracy and ToolParameterAccuracy work at the tool-call level. They distinguish choosing the correct operation from calling that operation correctly.

That separation produced an instructive result in AWS’s deliberate regression test. The authors changed the system prompt so the agent always returned an unhelpful refusal.

GoalSuccessRate fell to 0.0. Correctness also scored 0.0, and ToolParameterAccuracy scored 0.0.

ToolSelectionAccuracy still scored 1.0. The agent could apparently identify an appropriate tool even while failing the overall request.

This is why a single aggregate measure can conceal problems. Different evaluators capture independent dimensions, and one passing score cannot cancel every failed behavior.

After the authors restored the intended system prompt, GoalSuccessRate returned to 1.0. Correctness reached 0.9, while both tool measures reached 1.0.

All four then cleared the sample threshold of 0.8. GitHub unblocked the pull request.

AgentCore also accepts reference inputs for tests that require expected behavior. These inputs can include an expected response, natural-language assertions, or an expected tool trajectory.

A trajectory describes the sequence of tools an agent should call. AWS provides exact-order, in-order, and any-order evaluators for different workflow constraints.

An exact-order test fits a process where each step depends on the last. An any-order test fits independent retrieval operations whose sequence does not affect correctness.

The service also supports custom evaluators. Teams can define their own instructions, judge model, and scoring scale for domain-specific requirements.

Code-based evaluators provide a more deterministic option. They invoke an AWS Lambda function that can check schemas, patterns, keywords, or business rules.

That combination matters because not every requirement needs a model judge. JSON validity, forbidden fields, maximum call counts, and exact identifiers can often use ordinary code.

According to the custom evaluator guide, evaluators can operate at session, trace, or tool-call level. Custom logic can use a model or a Lambda function.

A practical gate should match evaluator type to requirement. Model-based scoring fits semantic qualities such as relevance, task success, or faithfulness.

Deterministic code fits binary constraints. Tool trajectory checks fit expected workflow structure.

The best signal comes from combining these methods. A response can be semantically correct while violating a schema, and a valid schema can still contain an irrelevant answer.

Ground truth also deserves careful use. Not every acceptable answer has one exact wording, especially for summarization or open-ended analysis.

Assertions can express the facts or behaviors that must appear without forcing a single response. Expected trajectories can specify tool behavior independently from prose style.

This turns the test set into more than a collection of prompts. Each scenario can state what success means at the response, task, and tool levels.

A Passing Score Still Has Blind Spots

The gate catches defined regressions, but its reliability depends on test coverage, score stability, identity design, and operational timing.

AWS says the complete pipeline takes about 10 minutes. Deployment, runtime startup, trace propagation, and model-based evaluation account for that duration.

That timing is acceptable for many pull requests. It is too slow for a local pre-commit check and may become costly in repositories with frequent updates.

Trace availability adds uncertainty. AWS observed propagation delays of 30 to 90 seconds after agent invocation.

The reference script retries every 30 seconds for as long as 10 minutes. Querying CloudWatch immediately can therefore produce an apparent failure before the trace exists.

Runtime architecture creates another implementation hurdle. AgentCore Runtime requires ARM64 container images, while standard GitHub-hosted runners use x86-64 processors.

The workflow uses QEMU and Docker Buildx for cross-platform image construction. That requirement increases build time and introduces another possible failure point unrelated to agent quality.

The most consequential limitation comes from model-based judgment. An LLM evaluator can assign slightly different scores when it assesses the same trace more than once.

A strict cutoff can therefore create flaky pull requests near the threshold. One run may pass at 0.81, while another fails at 0.79 without a meaningful code difference.

AWS recommends leaving margin between the acceptance threshold and the desired reliability target. Teams should also measure evaluator variance before making a check mandatory.

Repeated runs can estimate that variance, but they also increase evaluation usage. The reference example uses four evaluators across five prompts, producing 20 judge-model calls for one pull request.

The source does not assign a price to those calls. It does advise teams to monitor Bedrock usage as repositories and datasets grow.

Test coverage presents a familiar but sharper problem. A quality gate only measures scenarios included in its dataset.

Four or five convenient prompts cannot represent every production request, authorization state, failure mode, language, or adversarial input.

The sample’s machine token can access every tool. As a result, the end-to-end run does not prove that role-gated tools remain protected from ordinary users.

A broad score can also hide uneven performance. An overall average may pass while one critical scenario fails.

High-risk tasks should have scenario-level requirements. A payroll action, account deletion, or personal-data lookup should not borrow success from several easy arithmetic prompts.

Metrics also need ownership. Developers must know who approves threshold changes, adds new scenarios, and investigates evaluator disagreements.

Otherwise, a failing gate can create pressure to lower the threshold. The organization then preserves delivery speed by weakening the measurement.

False confidence is another risk. A green check shows that the tested version cleared specified criteria at one moment.

It does not certify every response, eliminate prompt injection, or validate infrastructure outside the deployed test stack. It also does not replace security review.

Production telemetry remains essential because users will find combinations that a curated dataset missed. AgentCore’s online evaluation can sample live sessions and publish results to CloudWatch.

The results documentation says online scores appear as CloudWatch metrics. Detailed evaluation events are stored in dedicated log groups.

Batch evaluation can then compare larger windows before and after model, prompt, or tool changes. That comparison provides a stronger baseline than one pull-request run.

A mature system should connect these stages. Production failures should become new evaluation scenarios, and repeated CI failures should inform agent design.

The quality gate is most credible when its dataset evolves with incidents. A static benchmark eventually rewards familiarity with the test instead of dependable behavior.

What to Watch After the First GitHub Actions Gate

The next test is whether teams can make these gates stable, security-aware, and representative enough to become required checks.

The first signal is adoption of ground-truth and deterministic evaluations in pull-request workflows. The four built-in metrics provide a useful starting point, but they do not express every business rule.

Watch whether teams add expected responses, assertions, and tool trajectories for high-value scenarios. Increased use of Lambda-based checks would show that agent evaluation is becoming ordinary software verification.

That development would strengthen the case for Amazon Bedrock AgentCore evaluation. It would reduce reliance on one model judge and make some failures easier to reproduce.

The second signal is role-aware CI coverage. AWS openly identifies the machine-token limitation, since its chosen flow bypasses user-role checks.

Teams should publish patterns that test distinct user identities without creating fragile refresh-token operations. Service accounts, direct authorization tests, and isolated test tenants are plausible routes.

If role-specific checks become common, the architecture will move closer to full end-to-end assurance. If they remain separate or absent, a green agent score will say little about authorization correctness.

The third signal is the connection between pull-request gates and production results. A small development dataset cannot remain representative without feedback from real sessions.

Watch for workflows that promote low-scoring production traces into reviewed regression cases. Online monitoring should discover failures, while curated CI tests prevent those failures from returning.

AWS’s dataset evaluation tooling is another relevant development. Its dataset runners can invoke agents, wait for telemetry, and evaluate scenarios through a coordinated process.

Broader adoption would reduce the custom orchestration now visible in the reference workflow. It would also make larger, versioned evaluation sets easier to maintain.

For developers, the practical question is no longer whether agent responses should be tested. It is which behaviors must block a merge, which need monitoring, and which require deterministic enforcement.

Start with failures that would cause a rollback, security incident, or broken business workflow. Give each one an explicit scenario and an evaluation method suited to its risk.

Then deliberately break the agent and confirm the gate fails for the expected reason. A check becomes trustworthy only after it catches a known regression.

Amazon has supplied a credible path from prompt tests to required GitHub status checks. The next step belongs to engineering teams: define the behavior they refuse to ship, encode it, and keep that definition current.