OpenAI Says Two API Settings Tripled GPT-5.6 Sol's ARC-AGI-3 Score

OpenAI changed two API settings and nearly tripled GPT-5.6 Sol's ARC-AGI-3 public score, from 13.3% to 38.3%. The OpenAI how account centers on retained reasoning and compaction, two context-management choices available through the Responses API. Together, OpenAI says they also reduced output tokens by a factor of six.

That result complicates the usual interpretation of an AI benchmark. The tested model did not receive new training, a larger reasoning budget, or a game-specific tool. Instead, OpenAI changed the harness, meaning the software layer that passes observations, actions, and conversation history between the model and benchmark.

The central conflict is therefore not GPT-5.6 Sol against another model. It is a generic, standardized harness against a model-aware production harness. One favors comparability and exposes agents to the same basic interface. The other preserves the state that OpenAI says its reasoning models need for long, interactive work.

This distinction matters because ARC-AGI-3 does not ask for one final answer. It presents unfamiliar games without natural-language instructions. Agents must infer each game's rules, discover its goal, test hypotheses, remember failures, and adapt over many actions.

A system that forgets its reasoning after every move faces a different task from one that can preserve an evolving plan. Likewise, an agent that loses its oldest observations when context fills up does not experience the same game as one receiving a compact summary.

OpenAI's result does not settle which harness produces the fairest leaderboard. It does show that harness design can change measured capability by more than many model upgrades. That puts pressure on benchmark operators, model developers, and enterprise evaluation teams to document the entire tested system.

OpenAI How the Score Jump Happened

The model stayed the same, while its memory pathway changed.

OpenAI published its analysis on July 29, 2026, after investigating unexpectedly weak GPT-5.6 Sol results. Its initial concern was straightforward. A model that had handled difficult mathematics and long game sessions appeared ineffective in simple-looking, two-dimensional environments.

The original discrepancy involved several score figures. OpenAI noted that GPT-5.6 Sol had received a 7.8% overall ARC-AGI-3 score, while GPT-5.5 scored 0.4%. On the public task set used for its controlled comparison, GPT-5.6 Sol scored 13.3% with the official harness.

OpenAI then rebuilt the evaluation through its Responses API. The revised harness retained reasoning between actions and replaced rolling truncation with compaction. According to the company's benchmark analysis, that configuration increased the public-set score to 38.3%.

The score uses Relative Human Action Efficiency, or RHAE. This metric compares successful progress with the number of actions humans need. An agent receives less credit when it solves a task inefficiently, so blind exploration and repeated mistakes carry a measurable cost.

OpenAI estimated that the average human tester scored 48% on the same basis. That estimate came from official gameplay logs, not a claim that every human receives 48%. ARC Prize separately describes 100% as the score for beating every game as efficiently as its human baseline.

The 38.3% result therefore did not close the human gap. It did move GPT-5.6 Sol much closer to OpenAI's estimated average tester. More importantly, it did so without a model update.

The output-token finding adds another reversal. Longer memory might sound more expensive because the agent carries more context. Yet OpenAI reports that the revised configuration used six times fewer output tokens. The model spent less output on repeatedly reconstructing rules and plans that it had already developed.

A game replay offers a concrete example. OpenAI showed one environment where no frontier model on the relevant leaderboard completed beyond the first level. Its revised Responses API harness reportedly allowed GPT-5.6 Sol to finish all six levels.

ARC Prize's verified results provide useful context. Its GPT-5.6 scorecard lists Sol at max reasoning with a 7.78% semi-private ARC-AGI-3 score. It also records an 87% result on the public game FT09, the first public ARC-AGI-3 game won by a model.

Those figures describe different sets and evaluation views, so readers should not combine them into one performance number. The controlled 13.3% to 38.3% comparison is the relevant evidence for OpenAI's two-setting claim.

That distinction is easy to miss in benchmark coverage. A public-set experiment, a semi-private verified score, and a result on one game answer different questions. OpenAI's experiment specifically asks how one model behaves when the harness preserves more of its working state.

The answer is striking but bounded. Retained reasoning and compaction improved this model on this public task set under OpenAI's implementation. The result does not establish an identical gain for every model, agent, context length, or production workload.

The Official Harness Was Testing Forgetfulness Too

ARC-AGI-3 measured the agent and the information architecture surrounding it.

ARC-AGI-3 differs from static benchmarks where a model receives a question and returns one answer. It consists of interactive environments that require repeated perception, planning, action, and correction. Its benchmark overview emphasizes goal acquisition, adaptable world models, long-horizon planning, and experience-driven learning.

The benchmark contains no natural-language instructions explaining each game. An agent receives a representation of the current frame and a set of possible actions. It must determine what objects matter, how actions change the environment, and what counts as progress.

This setup makes memory part of the evaluated capability. A useful hypothesis might emerge after several failed actions. A later observation can confirm or reject that hypothesis. If the harness deletes the reasoning connecting those events, the agent must rebuild its interpretation from a thinner record.

OpenAI found that the official harness discarded private reasoning after every game action. The model could still see earlier moves and brief notes. However, it could not access the plans, discoveries, or assumptions that produced those moves.

The harness also used rolling truncation. Once the conversation exceeded 175,000 characters, it removed the oldest messages. Long runs could therefore erase early observations and actions, even when those events remained relevant to the current level.

These behaviors were intentional consequences of a generic design, not evidence of a defective benchmark. ARC Prize favors a relatively simple harness because it exposes model limitations and supports comparison across providers. Commercial deployments often use provider-specific features that do not map cleanly across APIs.

That creates the article's main opponent: standardized evaluation versus production-aligned evaluation.

A standardized harness asks which model performs best through a common interface. This approach limits special treatment and reduces the chance that one provider's private optimization dominates the comparison.

A production-aligned harness asks how well a model performs when deployed as its creator recommends. This approach resembles real applications, but it can blur the boundary between model capability and system engineering.

Neither question is meaningless. They simply produce different measurements.

The concern arises when a leaderboard score is presented as a property of the model alone. If one harness repeatedly deletes state that another model expects to retain, the test also measures compatibility between model training and evaluation infrastructure.

ARC Prize has acknowledged how difficult the environments remain. Its technical report states that humans solved 100% of tested environments, while frontier systems scored below 1% as of March 2026. The benchmark was designed to expose a wide gap in adaptive efficiency.

GPT-5.6 Sol's later scores show progress, especially on individual public games. However, a 38.3% public-set result still leaves substantial room before human-level efficiency. It also comes from the model developer's own harness and analysis.

The lesson is not that ARC-AGI-3 became easy. It is that an interactive benchmark can accidentally test artificial amnesia alongside planning and exploration.

This issue extends beyond games. Coding agents must remember architectural decisions across tool calls. Research agents must retain which sources support or contradict a claim. Customer-service systems must preserve constraints introduced early in a conversation.

An enterprise team that evaluates those systems by repeatedly dropping their internal state may underestimate useful performance. A team that allows unrestricted provider-specific optimization may produce results that are hard to compare or reproduce.

The better response is fuller disclosure. A benchmark report should identify the model version, reasoning effort, context policy, retained response items, compaction behavior, prompt, tools, action budget, and scoring method. Without that information, a score lacks essential operational context.

Retained Reasoning Stops the Agent From Starting Over

Preserving reasoning gave GPT-5.6 Sol continuity across actions, reducing repeated interpretation work.

Reasoning models generate internal reasoning items before producing visible text or tool calls. These items are not the same as a user-facing explanation. They form part of the model's working state during a multi-step interaction.

In the official harness examined by OpenAI, each game action effectively ended that state. The next request included previous moves and short notes but discarded the reasoning that connected them. GPT-5.6 Sol repeatedly faced the same interpretive burden.

Imagine an agent exploring a grid with colored objects. It tests whether one color marks obstacles, then discovers that the same color changes after a successful move. That observation might overturn its first theory and establish a new rule.

A move log records what the agent clicked. It does not necessarily preserve why the agent clicked, which alternatives it rejected, or what the resulting frame implied. Those details can determine whether the next action advances a strategy or repeats an earlier mistake.

OpenAI used the Responses API and passed the previous response identifier between turns. Its current model guidance explains that GPT-5.6 can reuse available reasoning items across turns. Developers can also control how much prior reasoning remains relevant through reasoning-context settings.

According to OpenAI, retained reasoning changed GPT-5.6 Sol's behavior in two ways. The model spent less time thinking before each action, and it used more coherent strategies over longer runs. It no longer needed to reinterpret the game from the beginning after every move.

That explains how token use could fall while the retained state grew. Output tokens include the model's newly generated reasoning and responses. If earlier insights remain available, the model can avoid producing another lengthy reconstruction.

The distinction resembles the difference between a project team keeping decision records and receiving only a list of completed tasks. The task list shows what happened. It does not preserve the rejected options, constraints, or evidence that shaped the current plan.

For knowledge workers, the same problem appears when an AI assistant handles a long investigation. Each retrieved document changes the working theory. If the system preserves only links and short answers, later conclusions can lose the reasoning that made those sources relevant.

This is where an external searchable knowledge base can complement model context. Durable source records should remain inspectable even when an agent compacts its temporary working state.

Retained reasoning still introduces tradeoffs. Old reasoning can become stale after the task changes. An incorrect early theory can influence later steps if the system never marks it as rejected. More memory is not automatically better memory.

OpenAI's API guidance reflects that issue. It allows developers to preserve reasoning across all turns when goals remain stable or focus on the current turn when earlier reasoning has lost relevance. That choice should follow the workflow, not a blanket rule.

The ARC-AGI-3 games provide a favorable setting for continuity. The goal and environment remain related across actions, so earlier hypotheses usually matter. A support conversation that abruptly switches topics presents a different context-management problem.

Security and privacy policies also affect implementation. Some organizations cannot store response history indefinitely. Others use stateless or zero-retention configurations. Those teams must carefully manage the encrypted reasoning items and other response objects needed for continuation.

Retaining reasoning therefore describes a system behavior, not permission to expose private chain-of-thought content. Applications should preserve supported response items through the API while presenting concise, appropriate explanations to users.

OpenAI's experiment also suggests a practical evaluation question. If an agent fails after thirty steps, reviewers should inspect whether its plan was wrong or whether the harness removed the state needed to continue that plan.

Those failure modes require different fixes. A reasoning failure calls for a better model, prompt, or strategy. A state-management failure calls for different infrastructure. Treating both as model intelligence obscures where engineering effort belongs.

Compaction Preserves the Past Without Carrying Everything

Compaction replaced blunt deletion with a shorter representation of the history that still mattered.

Every long-running agent eventually encounters a context limit or an economic limit. Keeping every observation, tool result, and intermediate thought can make later requests slower and less focused. Simply retaining reasoning does not solve unbounded context growth.

The official ARC-AGI-3 harness handled growth through rolling truncation. Once history crossed its threshold, it removed the oldest content. That policy is predictable and provider-neutral, but it assumes old information matters less because it is old.

Interactive games violate that assumption. An early action might reveal the goal. A clue seen near the beginning might explain a symbol encountered much later. Deleting chronologically can erase foundational knowledge while retaining recent but repetitive frames.

Compaction uses a different policy. It condenses earlier context into a smaller representation that preserves relevant state. The agent can continue with a manageable context without carrying every original token.

OpenAI says its ARC implementation used a 175,000-token limit for compaction. The official harness used 175,000 characters for truncation. OpenAI argues these thresholds were broadly comparable because the action grids were tokenized at roughly one character per token.

That implementation detail matters. If the limits differed substantially, the score improvement might reflect a larger effective context rather than a better memory strategy. OpenAI's comparison attempts to hold the threshold close while changing how older information survives.

The company's compaction documentation describes compaction as a way to reduce context size during long Responses API workflows. The resulting compacted state can support continuation without requiring applications to resend an unlimited transcript.

Compaction is not lossless storage. A summary can omit details, flatten uncertainty, or preserve an incorrect interpretation. Its value depends on selecting the information that future actions will need.

ARC-AGI-3 creates a demanding test for that selection. The agent must preserve discovered rules, goals, object meanings, failed hypotheses, level progress, and planned actions. Repeated grids and redundant reasoning can receive less attention.

OpenAI reports that the compacted harness preserved learning across longer runs and generated fewer output tokens. Its comparison animation showed the revised agent thinking less per action and moving through games faster.

The sixfold output-token reduction is operationally important even without discussing monetary prices. Lower output volume can reduce latency and free capacity for more evaluations. It can also make agent traces easier to inspect, provided the compacted state remains sufficiently informative.

However, readers should treat the figure as workload-specific. ARC-AGI-3 produces repetitive text grids and many sequential actions. A legal review, medical workflow, or financial analysis may contain details that resist aggressive summarization.

Evaluation teams should test compaction quality separately from final task success. They can compare retained constraints, factual consistency, recovery after interruptions, and the agent's ability to explain which evidence shaped a decision.

A useful test introduces a critical fact early, fills the context with later activity, and then checks whether the agent applies that fact correctly. Another test changes a requirement midway and verifies that compaction preserves the revision instead of the original instruction.

These tests expose a core tradeoff. Rolling truncation offers transparent loss because reviewers know exactly which messages disappeared. Compaction offers selective continuity, but its omissions can be harder to detect.

For production agents, selective continuity often matches the task better. Humans do not remember every visual frame or sentence. They maintain a working model that prioritizes goals, constraints, discoveries, and unresolved questions.

That analogy should not excuse hidden failures. Developers need traces, replay tools, and controlled evaluations to see whether compaction preserved the right state. Otherwise, a confident final answer can conceal a missing early constraint.

The OpenAI how result is therefore less about making a context window larger. It is about changing what survives when the window cannot hold everything.

What the Tripled Score Does Not Prove

A developer-run public-set experiment cannot determine the fairest universal benchmark configuration.

OpenAI's evidence supports a narrow and valuable conclusion. GPT-5.6 Sol performed better in OpenAI's ARC-AGI-3 implementation when reasoning was retained and compaction replaced rolling truncation.

It does not prove that the official harness is invalid. The official setup serves a different goal: exposing model behavior through a generic interface that benchmark participants can reproduce across providers.

Provider-specific features complicate equal comparison. OpenAI can retain its native reasoning items through the Responses API. Another provider may represent internal state differently, expose different continuation controls, or rely on another memory architecture.

Allowing every company to optimize a private harness can turn a model benchmark into a systems competition. That format can still measure something useful, especially for buyers selecting complete agent systems. It no longer isolates model behavior as cleanly.

The result also comes from OpenAI's public task-set experiment. Public games are available for inspection and development, which creates more opportunity for harness tuning. Semi-private results provide a stronger check against direct task-specific adjustment.

ARC Prize's verified leaderboard still places GPT-5.6 Sol at 7.78% on the semi-private set at max reasoning. That score is ahead of other GPT-5.6 family variants listed there, but it remains far below human completion and efficiency.

The gap between 38.3% on OpenAI's public comparison and 7.78% on the verified semi-private evaluation needs careful interpretation. It does not automatically contradict the harness finding because the task sets differ. It does show why one public-set improvement should not become a general claim of human-like game learning.

OpenAI's 48% estimate for an average human tester adds context but not equivalence. Humans and models can fail differently, use different action patterns, and respond differently to unfamiliar interfaces. RHAE compresses those behaviors into a score.

The experiment also combines two changes. Retained reasoning and compaction were both active in the headline comparison. OpenAI describes the behavioral effect of each, but the published headline figures do not fully isolate each setting's contribution.

A more complete ablation would report four configurations: neither setting, retained reasoning alone, compaction alone, and both together. Repeated runs would help quantify variance, especially when agent exploration can follow different paths.

The output-token comparison deserves the same scrutiny. Fewer tokens indicate improved efficiency under this harness. They do not reveal total latency, input-token processing, compaction overhead, or the engineering work needed to maintain state.

Benchmark operators now face pressure from both directions. If they keep generic harnesses, they should explain that leaderboard results can understate production configurations. If they accept optimized harnesses, they should disclose which capabilities came from the model and which came from surrounding code.

Model providers face a parallel responsibility. They should publish reproducible configurations, prompts, context thresholds, action budgets, and scorecards. A claim about settings becomes more useful when independent teams can repeat it.

Enterprise buyers should avoid choosing between generic and optimized tests. They need both.

A generic test shows how candidate models behave under common constraints. A production-representative test shows whether the complete system succeeds in the buyer's actual workflow. The difference between those results measures integration sensitivity.

That sensitivity is itself a business risk. A model that performs well only through one specialized state-management stack may be harder to migrate. A generic harness that strips essential capabilities may push teams toward a weaker model for the wrong reason.

The relevant question is not whether the benchmark or OpenAI is correct. The question is which system boundary each score describes.

Three Signals to Watch After OpenAI's ARC-AGI-3 Result

Independent replication, semi-private testing, and better harness reporting will determine whether the result changes evaluation practice.

The first signal is an independent reproduction of the 13.3% to 38.3% gain. A credible replication should use the same public games, reasoning effort, action limits, context thresholds, and scoring method. It should also report run-to-run variation.

Replication would strengthen OpenAI's claim that state handling explains much of the observed performance gap. A substantially smaller gain would suggest that undocumented implementation details, prompting, or evaluator choices contributed more than the two named settings.

The second signal is a semi-private evaluation using retained reasoning and compaction. Public tasks help developers diagnose behavior, but unseen or restricted environments offer a stronger test of generalization. Similar gains there would show that the harness improvement transfers beyond familiar public games.

A small semi-private gain would weaken the broad interpretation. It might indicate that better memory helps the model execute known strategies without solving the deeper challenge of acquiring new goals and rules.

The third signal is a shift in benchmark disclosure standards. Future leaderboards should publish context-management policies beside scores. Reports should state whether reasoning items survived between actions, how old history was removed or compacted, and which provider-specific features were enabled.

This reporting would let readers compare two useful views. One view measures models through a standardized interface. The other measures complete systems configured according to provider guidance.

Developers should apply the same discipline to their own evaluations. Preserve the baseline before changing settings. Then test retained reasoning, compaction, and their combination separately on representative workloads.

Measure more than final success. Track output tokens, input growth, latency, repeated actions, constraint retention, recovery from failed hypotheses, and errors caused by stale memory. A higher score can hide new failure modes if the evaluation watches only the endpoint.

Teams should also inspect where continuity helps. Long coding tasks, interactive research, browser agents, and multi-stage analysis often depend on accumulated discoveries. Short classification or extraction requests may gain little from persistent reasoning.

The OpenAI how finding offers a clear warning: an agent evaluation can punish the model for memory policies imposed by the evaluator. It can also flatter the model through infrastructure that competitors cannot reproduce.

The next useful step is not to declare either harness correct. It is to run both, label them accurately, and publish enough detail for others to understand the difference.

For anyone building a long-running agent, the immediate question is practical: does your system preserve the goals, evidence, and rejected hypotheses that the next action needs? Test that continuity before blaming the model, and test it again after every change to compaction or context policy.