Sakana AI Reports Recursive Self-Improvement Gains, but Oversight Remains a Constraint

Sakana AI has completed two training cycles of recursive self-improvement through collective intelligence, reporting stronger research performance despite relying on one shared model. Its new Multi-Agent Self-Supervision method, called MASS, turns coordinated agent work into training data for that same model. The result is not an autonomous intelligence explosion. It is a controlled experiment in whether AI teams can create useful lessons for their shared underlying model.

The central conflict lies inside the loop. MASS asks one model to solve tasks, organize its agents, evaluate their work, and learn from the selected results. That design reduces dependence on stronger external supervisors. It also allows one blind spot to influence every stage.

The experiment starts with Qwen3.6-27B and focuses on open-ended research tasks involving finance, robotics, and pharmacy. After two rounds, the resulting model improves on several related research benchmarks. Performance falls slightly on two software engineering benchmarks, however, revealing an important boundary around the results.

That mixed outcome separates MASS from broader claims about self-improving AI. Sakana AI has produced evidence that collective trajectories can train a model more efficiently within a defined research setting. It has not shown that recursive improvement continues indefinitely, transfers everywhere, or remains safe without independent oversight.

MASS Turns Teamwork Into a Training Signal

The important change is that multi-agent coordination no longer ends when an answer is produced.

Most multi-agent systems use several model instances during inference. An orchestrator divides a task, assigns roles, collects results, and assembles a final response. The extra coordination can improve an answer, but the underlying model weights remain unchanged.

MASS carries that work into post-training. Its agents generate full trajectories containing messages, tool calls, intermediate artifacts, and results. The system ranks those trajectories, selects the strongest examples, and fine-tunes the shared model on them.

The researchers call this multi-agent self-supervision. The same language model acts as executor, workflow optimizer, and evaluator. Each role uses the same underlying weights, although separate model instances can handle different parts of an execution.

This distinction matters. The system does not combine several independently trained models with different knowledge or objectives. It asks copies of one model to organize themselves, then distills their selected behavior back into their common weights.

The process contains two connected loops. The inner loop searches for a better workflow while keeping model weights fixed. The outer loop trains the model on trajectories produced by the selected workflow.

A workflow is a prompt-based plan for organizing the team. It specifies agent roles, instructions, required outputs, and the order of communication. MASS can revise that plan without modifying the model during the search.

For every training task, the system keeps a champion workflow. A candidate team executes the task, and the model compares its workspace with the champion’s output. The optimizer then uses that judgment to revise the workflow.

Once the system finds a preferred workflow, it runs that workflow 18 times for each task. The model ranks the resulting workspaces through pairwise comparisons. A Bradley-Terry model converts those preferences into a score for each workspace.

MASS retains up to 16 trajectories per task. Fifteen can supply training examples, while one is reserved for validation. The workflow text is removed from the orchestrator’s initial training prompt before fine-tuning.

That removal has a specific purpose. It encourages the model to infer how work should be organized from the task itself. The researchers are not merely teaching it to repeat a fixed team structure.

The outer loop uses low-rank adaptation, or LoRA, which updates a limited set of model parameters during fine-tuning. Both orchestrator and subagent conversations become training targets. The updated model then returns as executor, optimizer, and evaluator for the next cycle.

The MASS paper was submitted on October 8, 2026, and describes two complete cycles. Sakana AI also released an open implementation containing task prompts, training configurations, evaluation tools, and pipeline documentation.

The release makes the core experiment inspectable. It also makes its scope unusually clear. MASS is a research pipeline built around selected tasks and controlled model updates, not a deployed system rewriting itself continuously.

Recursive Self-Improvement Through Collective Intelligence Produces Uneven Gains

MASS improves most where its evaluation tasks resemble the research work that generated its training data.

The experiment begins with Qwen3.6-27B inside a coding-agent environment. Its synthetic suite contains 12 research tasks across finance, robotics, and pharmacy. Each task requires code, quantitative analysis, and a written report.

Nine tasks were designated for training, although eight ultimately contributed training data. Three tasks, one from each domain, were held out from fine-tuning, validation, and checkpoint selection.

External judges measured progress without participating in the MASS training loop. Sakana AI says GPT-5.5 and Claude Opus 4.8 compared completed workspaces. Their judgments provided an outside evaluation of a process otherwise guided by the Qwen model.

On the three held-out synthetic tasks, the first-round model won 53.9 percent of comparisons against the base model. The second-round model reached 69.9 percent. It also won 60.8 percent of comparisons against the first-round model.

Training-task results were stronger. The first-round model recorded a 65.6 percent win rate against the base, while the second round reached 82.5 percent. That gap between training and held-out performance is expected, but it remains important.

Public benchmarks provide a broader test. Sakana AI reports that the second-round model achieved between 1.2 and 1.6 times the base model’s score per output token. Those gains appeared across MLR-Bench, DSBench, ScienceAgentBench, and AstaBench E2E-Hard.

The underlying scores tell a more detailed story. MLR-Bench rose from 1.71 for the base model to 2.25 after two cycles. Its paper-delivery rate increased from 57 percent to 76 percent.

ScienceAgentBench improved from 28.8 percent to 31.7 percent. AstaBench rose from 0.053 to 0.067, while its report-delivery rate increased from 50 percent to 76 percent.

DSBench moved only slightly, from 0.473 to 0.482. That small absolute change sits behind a stronger score-per-token ratio because the efficiency calculation also accounts for output length.

Two software engineering benchmarks moved backward. Terminal-Bench 2.0 fell from 0.375 to 0.352. SWE-bench Verified declined from 0.626 to 0.611.

Their score-per-token ratios reached 0.96 and 0.94 times the base model’s results. Those declines directly challenge any claim that MASS produced a generally stronger model.

The likely explanation is task alignment. The training suite emphasizes open-ended research with code, evidence, and reports. It does not include software engineering tasks resembling repository debugging or terminal operations.

Sakana AI suggests that adding software engineering tasks may address the gap. That remains a hypothesis until another cycle tests it. Expanding the task mix could also create interference between capabilities, rather than uniformly improving them.

The public benchmark design adds another qualification. MASS reports efficiency as score per output token, which is useful for evaluating productive reasoning. It does not include every cost required to create those improved outputs.

Workflow search, repeated executions, model comparisons, data generation, and evaluation all consume computation. The reported supervised-token totals also exclude input tokens and several upstream expenses.

The result is meaningful, but narrower than a simple efficiency headline implies. MASS appears to create denser training examples. Whether it reduces total development cost requires a complete accounting of the search and selection pipeline.

The Real Mechanism Is Information Flow, Not More Agent Personas

MASS gains appear to come from restructuring information exchange, not assigning theatrical identities to many agents.

Multi-agent systems often start with roles such as researcher, critic, engineer, or editor. These labels help divide work, but they can remain generic. A role called “careful analyst” carries little task-specific information.

MASS searches four workflow components: roles, instructions, contracts, and hops. A contract defines what evidence or artifacts an agent must return. A hop specifies when information moves between agents or returns to an earlier stage.

In one finance workflow, separate agents handle data preparation, feature construction, modeling, backtesting, auditing, reconciliation, and reporting. The data agent must report whether future information leaked into historical decisions.

That contract addresses a concrete failure mode. A financial backtest can appear successful when the model accidentally uses future data. Requiring explicit checks gives later agents evidence they can inspect before accepting the result.

Sakana AI analyzed how workflow components changed during search. It found that task information increasingly concentrated in contracts and communication order. Roles and general instructions carried less of that information over time.

The sample behind this analysis is small, covering eight or nine tasks depending on the comparison. Its mutual-information estimates describe workflow text, not every message exchanged during execution. The pattern should therefore be treated as a research clue.

Even with that limitation, it offers a useful account of Sakana AI MASS. The system improves by discovering what each agent must establish, what it must pass onward, and when another agent should inspect it.

Earlier multiagent debate research showed that model instances can improve reasoning by exchanging answers and critiques. MASS extends that idea from temporary inference behavior into supervised training.

The method also trains two different forms of behavior. Orchestrator trajectories show how to decompose broad work and coordinate specialists. Subagent trajectories show how to complete bounded assignments with clear output requirements.

That combination appears denser than training on long single-agent traces. In Sakana AI’s comparison, one multi-agent student reached a 68.3 percent win rate after processing 28.8 million supervised tokens.

The largest single-agent student reached 64 percent after processing 42 million tokens. By that measure, the multi-agent model achieved a higher result with 31 percent fewer supervised tokens.

The comparison does not isolate delegation as the sole cause. The groups used different workflows and verification instructions. Team executions also generated about 2.4 times as many tokens on average before training examples were selected.

External judges preferred multi-agent work in 85.1 percent of comparisons with single-agent outputs. Among ten pairs with similar trace lengths, team outputs still won 80 percent of comparisons.

Those results suggest that selection quality matters alongside team structure. MASS does not learn from every agent exchange. It creates many candidates, lets the model rank them, and trains on the preferred subset.

The mechanism resembles a filter that converts expensive collective work into concentrated lessons. It trades additional search and evaluation costs for better supervised examples.

That tradeoff has practical implications for developers. A useful AI team needs more than several copies of the same prompt. It needs explicit evidence contracts, review stages, and information-routing rules tied to the task.

The same principle applies when humans coordinate AI-assisted research. A searchable knowledge base helps only when contributors preserve evidence, context, and decisions in forms that others can inspect.

MASS automates part of that organizational design. Its deeper claim is that coordination itself can become learned capability. The model does not just perform better work through a temporary team. It absorbs patterns from the team’s successful behavior.

One Model Still Judges Its Own Homework

The defining strength of multi-agent self-supervision is also its largest unresolved risk.

MASS uses one shared model as task solver, workflow optimizer, and in-loop evaluator. That homogeneous design tests whether a model can improve without depending on a permanently stronger teacher.

It also creates correlated failure. If the model misunderstands a task, each agent may reproduce that misunderstanding. The evaluator can then reward the same error that the executor produced.

Repeated cycles raise the stakes. A flawed trajectory can be accepted as superior, selected for training, and incorporated into the next model. The updated evaluator may become more confident in the shared mistake.

Sakana AI openly identifies shared blind spots as a central safety problem. Its project overview argues that self-evaluation must remain open to correction as capabilities grow.

The external judges help measure this experiment, but they do not solve the deployment problem. GPT-5.5 and Claude Opus 4.8 remained outside the training signal. Their availability therefore provided a check that a truly autonomous loop might lack.

MASS reports that agreement with those external judges increased from 73 percent to 93 percent across three model generations. That is evidence that the internal evaluator became more aligned with stronger judges on these tasks.

Agreement is not equivalent to correctness. Two evaluators can share assumptions, reward polished but flawed reports, or miss errors requiring real-world experiments. Open-ended research makes that distinction particularly important.

The benchmark suite is also compact compared with the range implied by recursive self-improvement. Only three synthetic tasks were held out from post-training. Two complete cycles provide evidence of iteration, but not indefinite recursive progress.

Community skepticism has focused on this gap. Some Hacker News commenters argued that the work looks closer to improved agent coordination than unrestricted recursive self-improvement. Others questioned why an apparently recursive process stopped after two rounds.

That criticism identifies a real naming problem. Strict recursive self-improvement often implies a system that autonomously changes the mechanisms responsible for creating its successors. MASS instead alternates human-designed workflow search with supervised fine-tuning inside a fixed experimental pipeline.

The model influences workflows, training data, and evaluation decisions. Humans still define the architecture, task suite, optimization method, benchmark protocol, and cycle boundaries. They also provide the computing infrastructure.

Calling MASS recursive is defensible because each updated model helps generate and select the next model’s training data. Calling it open-ended self-improvement would overstate what the experiment establishes.

Optimizer quality presents another constraint. Sakana AI tested configurations that inserted GPT-5.5 into the workflow-search loop while keeping the base Qwen model as executor.

Using GPT-5.5 only as evaluator produced a limited improvement. Using it as both evaluator and optimizer found winning workflows for all 12 tasks within five search iterations.

The fully homogeneous loop eventually found winning workflows for 11 of 12 tasks, but required as many as 13 iterations. Sakana AI interprets this as evidence that the optimizer can become a larger bottleneck than the evaluator.

A good critique is useless when the optimizer cannot convert it into a better process. This result shifts attention from scoring accuracy toward the model’s ability to revise complex workflows.

Search stability remains weak. Sakana AI’s workflow results fluctuate, with strong iterations followed by sharp regressions. Reaching a good workflow does not guarantee that the next revision preserves it.

These limits make the result more interesting, not less. MASS exposes where a homogeneous improvement loop fails under observation. It provides measurements for transfer, judge agreement, optimizer strength, and workflow instability.

Collective Intelligence Does Not Remove the Oversight Problem

A team of identical models can broaden a search, but it cannot guarantee independent judgment.

Collective intelligence usually gains strength from differences among participants. Human teams bring separate experiences, incentives, expertise, and institutional roles. Those differences can expose errors that one person would miss.

MASS uses multiple instances of one shared model. Random sampling and specialized assignments create behavioral variation, but every agent inherits the same weights and much of the same training history.

That architecture can still produce useful diversity. Separate agents explore different approaches, work with different intermediate evidence, and critique one another. However, their disagreement does not establish genuine independence.

The problem becomes sharper on non-verifiable tasks. Sakana AI uses that term for work without an automatic rule-based success signal. A research report can be coherent and well supported while still containing an unnoticed conceptual error.

A model evaluator may favor outputs that match its own style or reasoning habits. Training on those selections can strengthen the evaluator’s preferences without strengthening contact with reality.

This is a broader scalable-oversight problem. As agents perform work that humans struggle to assess, developers need weaker supervisors to monitor stronger systems. More capable model judges help, but they also move evaluation authority away from humans.

OpenAI’s monitorability research finds that reasoning traces can reveal more about agent behavior than actions and final answers alone. It also describes monitorability as incomplete and dependent on both the agent and monitor.

That matters for MASS because its training examples contain rich trajectories. Messages, tool calls, and intermediate artifacts create more opportunities for inspection. They also create a larger surface where subtle errors can hide.

Directly optimizing visible reasoning can introduce another danger. A model may learn which explanations earn approval without changing the behavior that produced a failure. Oversight then rewards legible rationalization instead of reliable work.

Sakana AI does not claim to have solved this problem. Its paper warns that the same weakness can compromise a model’s behavior and the mechanism intended to correct it.

Independent checks therefore remain necessary. These can include external models, human experts, executable tests, trusted data pipelines, adversarial evaluations, and delayed real-world validation.

No single check covers every failure. Code can run while implementing the wrong objective. A report can cite real evidence while omitting contradictory results. Several models can agree because they share training biases.

The strongest near-term use of MASS may be bounded research with verifiable intermediate steps. Financial pipelines can test chronological data splits. Robotics experiments can check simulation outputs. Pharmacy analyses can enforce data and calculation contracts.

The more subjective the final objective becomes, the weaker self-judgment becomes as a sole signal. Strategy, scientific novelty, policy advice, and safety analysis all depend on values or evidence that model consensus cannot settle.

Anthropic’s discussion of recursive self-improvement similarly places oversight, validation, and verification at the center of future AI development. The common pressure is clear even when technical approaches differ.

If AI contributes more of the work needed to improve AI, supervision cannot remain an afterthought. Evaluation systems must advance alongside capability systems. MASS supplies a useful test bed for studying that race.

The Next Three Signals Will Decide What MASS Actually Proves

The next phase must test duration, transfer, and correction rather than repeat the strongest headline result.

The first signal is whether improvement survives additional cycles. Two rounds establish recurrence, but they do not establish sustained recursive self-improvement through collective intelligence.

A third and fourth cycle should measure whether gains continue, plateau, or reverse. Researchers should also report how workflow-search costs change as the model improves.

Continued gains with stable external evaluation would strengthen the recursive claim. A plateau would suggest that the model exhausts the available lessons within its task distribution. Regression could reveal accumulated judge error or training interference.

The second signal is transfer into domains excluded from the original research suite. The declines on Terminal-Bench 2.0 and SWE-bench Verified provide a demanding baseline.

Adding software engineering tasks should improve those benchmarks if the method learns transferable coordination. If gains appear only after direct domain-specific training, MASS will look more like an efficient specialization method.

That would still be valuable. It would simply narrow the claim from broad self-improvement to better task-conditioned post-training.

A convincing transfer test should also preserve performance in finance, robotics, and pharmacy. Otherwise, new capabilities may replace old ones rather than compound them.

The third signal is whether the system can detect and reverse a shared blind spot. Researchers could introduce hidden evaluation traps, conflicting evidence, or tasks where polished consensus leads to a wrong conclusion.

Success would require more than external judges catching the failure. The homogeneous loop should identify the problem, revise its workflow, and avoid training on the defective trajectories.

That test would connect capability claims with control. It would show whether collective intelligence can create corrective pressure when all agents begin from similar assumptions.

Organizations evaluating Sakana AI MASS should focus on these signals before treating it as a general recipe. The present results support experimentation with evidence-heavy, bounded research workflows. They do not support removing human review from consequential decisions.

The larger lesson is already visible. Better agent teams can generate better training data, and those lessons can move into model weights. Yet every gain depends on what the team counts as success.

Recursive self-improvement through collective intelligence becomes trustworthy only when the loop can recognize its own shared errors. Which independent checks would your organization require before letting such a system select the evidence used for its next training cycle?