OpenAI Models on Amazon Bedrock Just Got a Different Cost Test
OpenAI models on Amazon Bedrock received a new production benchmark on September 11, and its results challenge the cheapest-token rule. AWS and OpenAI tested five configurations across academic problems, research agents, professional documents, and latency measurements. The central finding was consistent. A model with cheaper tokens can still cost more when weak answers, repeated searches, and human rework enter the calculation.
The comparison covers GPT-5.6 Luna, Terra, and Sol, alongside GPT-5.4 Mini and Nano. Instead of declaring one universal winner, the benchmark study asks what each successful outcome consumes. That includes failed attempts, accumulated context, tool calls, latency, and deliverables that miss an acceptance rubric.
This reframes the buying decision for teams deploying agents and document workflows. The primary contest is no longer cheap tokens versus expensive tokens. It is posted token pricing versus the full cost of an accepted result. The new open-source harness gives developers a way to test that contest with their own tasks.
The benchmark replaces token rankings with outcome rankings
The benchmark changes the unit of comparison from generated tokens to work that passes a defined quality threshold.
That sounds like a small accounting adjustment. It changes which model appears economical.
AWS and OpenAI evaluated several workload shapes because a single accuracy test cannot represent a production system. Their academic suite included AIME, GPQA Diamond, and MMLU-Pro. Sample sizes ranged from 60 AIME problems to 198 GPQA Diamond questions, with 140 MMLU-Pro questions between them.
Each model incurred usage across successful and unsuccessful attempts. The evaluators divided total observed usage cost by the number of correct answers. This produced a cost per correct answer, rather than a cost for one request.
The distinction matters when accuracy differs sharply. GPT-5.6 Sol answered 75 percent of the sampled AIME problems correctly. GPT-5.4 Mini reached 37 percent. Sol also led Mini on GPQA Diamond, at 68 percent versus 43 percent, and MMLU-Pro, at 82 percent versus 59 percent.
A model reaching 37 percent accuracy would require about 2.7 independent attempts per success on average. Production retries are rarely independent, however. The same ambiguous prompt or missing evidence can push every attempt toward a similar failure.
Luna delivered the lowest observed cost per correct answer across the tested samples. That included comparisons with Nano, despite Nano carrying a slightly lower nominal token rate under the recorded assumptions. Luna used fewer billed tokens in the tested configuration and converted more attempts into accepted answers.
The result does not prove that Luna is always the least expensive model. It shows why the invoice cannot be inferred from the rate card alone. Prompt length, output length, reasoning settings, retry policies, and required accuracy all affect the final ranking.
The harness makes those dependencies visible. It records answers, token consumption, quality scores, and calculated outcome costs. Teams can inspect the underlying results instead of accepting a composite leaderboard score.
That transparency is important because model evaluations often compress several tradeoffs into one number. A production owner needs to know whether a model failed through factual error, missing structure, excessive turns, or output truncation. Each failure suggests a different response.
A factual miss might justify a stronger model. A structural miss might be fixed with a clearer rubric. Repeated searches can indicate weak tool selection or an inefficient agent loop. Truncation points toward output limits rather than reasoning quality.
For OpenAI models on Amazon Bedrock, the benchmark therefore establishes a more useful first question: what counts as success in this particular workflow? Only after defining that threshold can a team compare the resources needed to reach it.
Cost per correct answer exposes the retry penalty
Every incorrect answer belongs in the model-selection budget, even when the application quietly retries it.
Per-token comparisons usually assume that two models complete equivalent work. The academic results show why that assumption fails. Higher accuracy changes the expected number of calls, while token efficiency changes the size of each call.
Consider an application that answers technical questions before publishing them to customers. A wrong answer might trigger an automated retry, a fallback model, or a human review. None of those consequences appears in the initial token quote.
A cost per correct answer calculation captures the direct model usage from failed attempts. A more complete production calculation can add validation, review time, downstream corrections, and customer-facing risk. The appropriate boundary depends on who owns the workflow.
The AWS study deliberately uses observed samples rather than promising a universal ranking. That choice limits the scope of its conclusions, but it improves their practical value. The recorded tasks, prompts, settings, and scoring rules can be examined and changed.
Teams should preserve that discipline when adapting the harness. A known-answer set needs to resemble actual traffic. Easy questions can make every model look interchangeable, while unusually difficult questions can overstate the need for a premium model.
Failure costs also vary by use case. An imperfect internal summary might be corrected in seconds. An incorrect compliance statement can start a longer review process. Accuracy thresholds should reflect that difference before any model run begins.
This is where routing becomes more useful than a single corporate default. Routine classification can go to the model that clears a modest threshold efficiently. Difficult analysis can escalate after a validator detects uncertainty or failure.
Routing still needs measurement. A cheap first attempt followed by frequent escalation can cost more than sending the task to a stronger model immediately. It can also add latency and duplicate context across calls.
The benchmark results suggest that Luna deserves the first evaluation slot for many high-volume tasks. It produced the lowest observed outcome cost across the study’s academic samples. Yet Sol remained the stronger option when difficult-question accuracy served as a hard gate.
That is a workload decision, not a brand hierarchy. Luna, Terra, Sol, Mini, and Nano occupy different points across quality, speed, and consumption. Their names do not reveal which point meets a particular acceptance rule.
Reasoning settings complicate the picture further. The Amazon Bedrock comparisons disabled reasoning for the tested models, creating a deliberate cost floor. Enabling reasoning can improve results, increase usage, or do both.
A fair evaluation must treat each model and reasoning level as its own configuration. Comparing one model without reasoning against another at a higher reasoning setting obscures the mechanism behind the result.
The outcome framework also makes price changes less disruptive. A team can update current rates in its evaluation records and recalculate the ranking. It does not need to rebuild the quality study each time commercial terms move.
That separation between stable quality evidence and changing commercial inputs is valuable. It turns model selection into an operating process, rather than a one-time procurement decision.
Agent trajectory cost turns extra steps into compounding context
For research agents, the expensive behavior is often not one long answer but an unnecessary sequence of tool calls.
The study tested that problem with a 50-question, stratified DeepSearchQA sample. These were multi-step research questions handled through live web-search and page-fetching tools.
The agent used client-managed conversation history with storage disabled. Every new turn resent the system prompt, earlier tool results, and accumulated context. As the trajectory grew, each request became larger than the one before it.
This creates a compounding effect. Per-turn input grows roughly linearly when history keeps accumulating. Cumulative billed input can approach quadratic growth as the number of turns rises.
An eight-turn agent therefore does more than make three additional calls compared with a five-turn agent. Its later calls also carry more prior material. Each round trip adds latency while duplicated context increases consumption.
Mini averaged 7.6 turns per DeepSearchQA question, the highest count among the tested models. Many of those turns were additional search loops. Its average input volume reached 114,000 tokens per question, compared with 50,000 for Terra.
That is 2.3 times Terra’s input volume before considering answer quality. Mini recorded a mean F1 score of 0.39, while Terra reached 0.50. F1 measures overlap between expected and produced answer content, balancing precision with recall.
A response passed when its F1 score reached at least 0.7. The evaluation used a deterministic pre-check followed by a fixed GPT-5.5 autorater. Freezing the grader reduced one source of variation across the model runs.
Terra completed the research trajectories with fewer turns and better average quality than Mini. Its higher nominal token rate did not determine the final result because Mini repeatedly resent more accumulated evidence.
Luna recorded fewer turns than Mini and a substantially lower observed agent trajectory cost per passing answer. Its pass-level cost was about one eighth of Mini’s in this sample. Nano had cheaper nominal tokens but passed only 18 percent of questions.
The three GPT-5.6 configurations all produced higher mean F1 scores than the two baselines. That supports the article’s broader claim, but the sample remains limited. Fifty questions cannot settle close differences across every research domain.
The mechanism still deserves attention even if another workload reverses the model ranking. Any client-managed agent that resends history pays for its trajectory design. Better model behavior and better orchestration can both shorten that history.
A model might save turns by choosing a more relevant search query, recognizing sufficient evidence, or composing an answer without another lookup. An orchestrator can save turns by pruning tool output, summarizing history, or enforcing a search budget.
These improvements should be evaluated separately. Otherwise, teams might credit a model for an orchestration change or blame it for unnecessary context inserted by the application.
The Responses API provides an OpenAI-compatible request structure through Amazon Bedrock. Compatibility simplifies model substitution, but equivalent requests do not guarantee equivalent trajectories.
Tool schemas, stopping rules, prompt design, and regional behavior still affect the observed run. A useful test keeps these elements fixed while switching one model configuration at a time.
Teams should also log more than total tokens. Turn count, tool choice, retrieved bytes, repeated queries, completion status, and validator outcomes reveal why one trajectory costs more.
For agents, the practical metric is accepted answers per complete run. Agent trajectory cost then explains the difference between models that appear similar on a one-call rate card.
Professional deliverables make quality part of the invoice
Documents create costs after generation because a plausible draft can still fail the requirements that professionals must satisfy.
Many enterprise outputs cannot be graded by exact string matching. A compliance brief needs required caveats. A financial plan needs consistent assumptions. A care protocol must include specific safeguards and usable structure.
The study addressed that problem with 48 tasks drawn from GDPval. GDPval evaluates realistic knowledge-work deliverables using criteria created around professional tasks. The broader GDPval framework covers 44 occupations across nine sectors.
The AWS evaluation used human-authored rubrics and weighted their requirements. A document passed after earning at least 70 percent of available rubric points. That transformed subjective quality into an explicit acceptance gate.
All three GPT-5.6 configurations earned higher observed rubric scores than Mini and Nano with reasoning disabled. The largest reported differences appeared in law, nursing, and financial-advice tasks.
Those category findings are exploratory because each subgroup was small. They still illustrate why format and completeness belong in the benchmark. A response can contain correct facts while omitting the caveat that makes a professional document usable.
Luna scored higher than Mini on 31 of the 48 deliverables. It scored lower on nine and tied on eight. Luna passed 27 tasks, while Mini passed 20.
Nano passed 35 percent of the tasks. Mini reached 42 percent, and Luna reached 56 percent. Sol passed 31 of 48 deliverables, supporting its place when quality is a hard requirement.
These results alter the economic question. A lower model charge has limited value if employees must repeatedly restore missing sections. Review and rework can dominate the cost of generating the first draft.
Quality-gated document production therefore needs two connected measurements. The first is model usage per passing deliverable. The second is the human effort required to turn a failed or marginal output into an accepted one.
The benchmark directly measures the first. Organizations must supply the second from their own workflows. Review time can be collected through approval systems, editing logs, or structured evaluator feedback.
That extra evidence can change the preferred model. Terra or Sol might justify greater model consumption if their drafts require materially less professional review. Luna might remain preferable when its pass rate clears the business threshold with lower total usage.
Output limits introduce another uncertainty. The evaluation capped deliverables at 8,192 tokens. That truncated six Luna outputs, nine Terra outputs, seven Sol outputs, no Mini outputs, and one Nano output.
Those truncations count as real outcomes under the tested configuration. They also make it difficult to separate model quality from the imposed length ceiling. A higher cap might improve rubric completion while increasing consumption.
A careful replication should test both the cap and the model. It should also inspect whether longer documents gain useful content or merely repeat themselves. More output does not automatically mean a better deliverable.
Rubrics need similar scrutiny. A generic scoring form will miss domain-specific failure modes. Legal, clinical, financial, and engineering work require different evidence, qualifications, and escalation rules.
Teams can begin with 50 to 100 representative tasks, as the benchmark authors recommend. Each task should include a known-good result or an acceptance rubric that reviewers can apply consistently.
A useful set includes common work, difficult edge cases, and failure-sensitive examples. It should also preserve the input files and context that employees actually use. Sanitized toy prompts tend to understate retrieval and formatting problems.
For knowledge-heavy document teams, maintaining these evaluation sets becomes part of operational memory. A searchable engineering knowledge base can help preserve rubrics, reference files, and prior failure analyses.
The goal is not to eliminate professional judgment. It is to spend that judgment on representative evaluation and consequential exceptions, rather than reviewing avoidable defects in every generated draft.
OpenAI models on Amazon Bedrock still need local validation
The published rankings are evidence for what to test first, not permission to skip testing.
The harness provides a stronger model-selection method, but its own limitations prevent a universal recommendation. Sample composition, region, reasoning settings, output caps, and grader choices all influence the results.
The authors ran the latency comparison in July 2026. Luna and Terra used a single-region setup in AWS’s western United States region. Sol used an eastern region because its behavior and availability differed.
Across 12 matched configurations, median time to first token averaged 21 percent lower for Luna on Amazon Bedrock. Terra averaged 5 percent lower. Time to first token measures the delay before streamed output begins.
For outputs containing at least 500 tokens, Luna’s throughput averaged 43 percent higher on Amazon Bedrock. Terra’s advantage averaged 4 percent. These measurements compared the same models across two application paths.
The observed worst-case delay relative to the median ranged from 2.1 to 2.5 times on Amazon Bedrock. The corresponding range on the OpenAI API was 4.6 to 6.6 times.
Those maxima are not estimates of 99th-percentile latency. The study explicitly describes them as point-in-time observations. Shared infrastructure changes with region, load, quotas, routing, and request shape.
A production team should repeat the latency run from its own deployment region. It should test realistic concurrency, prompt sizes, streaming behavior, and service-level objectives. Median results alone can hide user-visible tail delays.
The quality tests also used one deliberate configuration choice. Reasoning was disabled, which created a lower-usage baseline. Applications depending on difficult planning or synthesis should repeat the runs at the reasoning levels they expect to deploy.
Grading introduces another concern. DeepSearchQA combined a deterministic step with a GPT-5.5 autorater. That process is reproducible, but any model grader can carry preferences or overlook domain-specific errors.
Human review remains useful for calibration. Reviewers can examine disagreements near the passing threshold, then adjust the rubric or deterministic checks. They should not change the rules after seeing which model wins.
GDPval has a different constraint. Its 48-task slice is broad enough to reveal patterns but too small for confident conclusions about individual professions. The reported category differences need larger, targeted samples.
Teams must also resist benchmark leakage. If prompts resemble public evaluations too closely, results may overstate real performance. Private tasks collected from actual workflows provide a better test of local value.
The open-source benchmark harness supports this adaptation. It includes scripts for academic evaluations, DeepSearchQA trajectories, GDPval deliverables, and performance comparisons.
Reproducibility does not eliminate operational variables. Model revisions, service updates, and prompt changes can move results. Evaluation records should therefore store model identifiers, dates, regions, settings, and rubric versions.
Security and governance requirements can also outweigh a small efficiency difference. OpenAI models became generally available through AWS in June 2026, using AWS-native controls and procurement workflows. The AWS availability matters for organizations already standardized on that environment.
Yet platform fit should remain separate from model quality. Bedrock can simplify governance without making every model suitable for every task. The harness helps teams test quality after infrastructure requirements narrow the available choices.
A sound decision therefore combines four gates. The model must satisfy governance rules, clear the quality threshold, meet latency expectations, and minimize the total resources consumed by accepted work.
No public benchmark can set those gates for an individual organization. It can only show which measurements reveal the hidden tradeoffs.
Three signals will show whether outcome pricing becomes standard
The next test is whether teams operationalize outcome measurement instead of treating this benchmark as another static leaderboard.
The first signal is adoption of private, workload-specific evaluation sets. Over the next several months, the most informative organizations will publish methodology details rather than universal model rankings.
A good evaluation set includes routine tasks, costly failures, and accepted reference outputs. It also records the retry policy, human review process, and production configuration.
If more teams report cost per accepted result, the benchmark’s core judgment becomes stronger. If most comparisons remain limited to token rates and public exams, outcome-based selection will remain a specialist practice.
The second signal is model routing driven by trajectory telemetry. Research agents should expose turn counts, duplicate searches, context growth, validator failures, and escalation frequency.
Routing systems can use that evidence to assign simple tasks to Luna and escalate difficult work to Terra or Sol. However, the routing policy must beat a single-model baseline after all failed first attempts are included.
If routing reduces accepted-result cost without weakening quality or latency, workload shape has become a practical pricing input. If escalation consumes the savings, simpler model assignments will remain more credible.
The third signal is repeated benchmarking after model, service, or commercial changes. OpenAI and AWS began a broader partnership in 2026, including OpenAI models and agent infrastructure on Bedrock. The Amazon partnership gives both companies reasons to keep adjusting their joint offering.
Each change can alter the preferred configuration. A revised model might use fewer tokens, follow tools more reliably, or improve long-document completion. A regional service update might change latency without affecting quality.
The original benchmark already demonstrates why reruns matter. Updated commercial terms changed outcome rankings without changing prior answers. Future model revisions can move both sides of the equation.
Teams should schedule reevaluation when a model version changes, a material rate update arrives, or production prompts shift. They should also rerun tests when observed failure patterns no longer resemble the original evaluation set.
The actionable starting point is modest. Select 50 to 100 known tasks, define a passing rule, and run each candidate under the intended production settings. Count failures, retries, turns, tokens, latency, and review effort.
Then calculate the resources consumed by accepted outcomes. Inspect the failure categories before choosing a winner. A model that looks economical on average might still fail the cases that carry the greatest business risk.
OpenAI models on Amazon Bedrock now have a public framework for making that decision. The durable insight is not that one GPT-5.6 variant wins every workload. It is that the cheapest token has no business value until the system turns it into acceptable work.
Before renewing a default model choice, ask one measurable question: how much does your complete workflow consume for every answer, trajectory, or deliverable that your organization can actually use?
