WorldWeaver Gives Multi-Agent Video Models a Shared Memory, but Minecraft Is Only the First Test

WorldWeaver has introduced an explicit shared memory into a two-agent video model, challenging pipelines that reconstruct the world from recent frames alone. The model, also called W², stores evolving information in world state registers while generating synchronized Minecraft views. Its results suggest that believable video is not enough for a persistent interactive world.

The work comes from researchers at UCLA and Adobe Research. They submitted the WorldWeaver paper on July 23, 2026. Its central contest is architectural: persistent shared state versus observation history compressed inside a rolling context window.

That distinction matters because multiple agents never see exactly the same scene. One player can walk behind a wall while another continues building outside. A model that remembers only visible pixels must reconstruct hidden changes when those views reconnect.

WorldWeaver instead maintains learnable tokens that summarize the shared environment and each agent’s status. It updates those tokens after every generated chunk. The next chunk then depends on both recent frames and the latest committed state.

The reported gains are substantial within the researchers’ two-player Minecraft evaluation. WorldWeaver reached an aggregate world score of 105.1, compared with 81.0 for a retrained Solaris baseline. However, the experiment remains narrow, simulator-based, and dependent on supervision that real environments rarely provide.

WorldWeaver Moves Shared State Inside the Generator

The consequential change is not another improvement to video memory, but a separate state pathway that persists across agents and rollout steps.

Streaming autoregressive diffusion models generate a video in consecutive chunks. Each chunk conditions the next one through recent frames and a key-value cache, which stores intermediate attention information. This design keeps generation manageable without reopening the entire history.

The limitation appears when an observation disappears from that recent window. The generator must infer the missing information again from whatever pixels remain. That reconstruction can introduce inconsistent locations, object states, identities, or relationships.

The problem grows in a multi-agent setting. Each camera provides a partial projection of one underlying environment. A valid model must reconcile those projections while tracking changes that occur beyond a particular camera’s view.

WorldWeaver adds world state registers, which are learnable tokens reserved for the environment’s evolving state. They sit beside the visual tokens but serve a different purpose. The register summarizes information that should survive when individual frames leave the active context.

At each rollout step, the generator reads its previous register, recent frames, and the current action. It then commits an updated register after producing the new visual chunk. That register conditions the following chunk.

The process creates an alternating sequence of observations and state updates. Recent frames provide detailed local evidence. The register provides a compact continuity mechanism across time and viewpoints.

This structure differs from simply extending the frame cache. A larger cache preserves more observations, but those observations remain camera-dependent and visually encoded. The model must still extract the relevant state whenever it needs that information.

WorldWeaver asks the model to maintain that abstraction continuously. The researchers describe the register as both persistent and dynamically updateable. Every agent contributes evidence to the shared representation as the rollout advances.

The approach also limits stale register history. During inference, the system retains the initial register and the newest committed register rather than accumulating every intermediate state. That choice makes the mechanism closer to recurrent state than an expanding memory archive.

The project’s method overview shows synchronized two-player Minecraft rollouts. Both players act inside one generated environment, while a hidden representation tracks their positions and the surrounding scene.

Minecraft is useful because the environment exposes precise simulator state. Researchers can measure agent positions, velocity, orientation, and controller inputs. They can also capture an overhead view unavailable to either player.

Those signals make hidden state inspectable. They let the team ask whether a generated video corresponds to the same underlying world across both cameras. Pixel quality alone cannot answer that question.

The news is therefore less about prettier Minecraft footage. It is about moving consistency from an expected side effect into a supervised component of generation.

Why Frame History Becomes a Liability for Multiple Agents

Observation history records what cameras saw, while an interactive world model must preserve what remains true after those observations disappear.

A single-camera video model can often conceal small logical errors. An object might shift slightly between distant frames, yet the sequence can still look plausible. Multi-agent generation makes those errors easier to expose.

Suppose Alpha places a block while Bravo looks in another direction. Bravo’s later view should contain that block when the camera turns back. Alpha’s action must alter one shared environment, not only Alpha’s video stream.

Now suppose the players separate. Their recent frame histories contain different terrain, objects, and camera movements. When they meet again, the model must reconcile both histories without inventing contradictory geometry.

A rolling visual cache provides weak support for this task. It favors recent observations and entangles state with appearance. Information about an unseen player or hidden structure can fade as new frames replace old ones.

An explicit state can preserve facts independent of the current camera. Position, orientation, nearby layout, and completed actions can remain available even when their source pixels leave the window.

WorldWeaver supervises its registers with three kinds of targets. Agent statistics capture position, velocity, and orientation. Bird’s-eye views constrain global layout, while scene text supplies a semantic description of current behavior.

Each target addresses a different failure mode. Agent statistics help preserve local motion and identity. Overhead representations give both agents a common geometric reference.

Scene text encourages the register to retain meaningful relationships rather than only low-level visual features. The paper’s captioning process describes each player’s behavior, relative position, distance, and viewing direction.

These annotations use synchronized agent views, overhead frames, and controller inputs. They provide unusually direct knowledge of what happened inside each generated chunk.

That supervision is important because the video-generation objective does not specify what the register should remember. An unsupervised token can absorb useful information, irrelevant texture, or a mixture that becomes unstable during rollout.

The researchers found that state targets changed both generation and interpretability. Training-only decoding heads can recover agent coordinates, overhead features, and scene descriptions from the register. Those heads are removed during inference, so they do not add decoding work to deployment.

This is also where the approach places pressure on existing streaming pipelines. Extending context windows improves access to prior observations, but it does not create a canonical shared state.

The comparison is especially relevant to Solaris, a February 2026 multiplayer world model. Solaris introduced synchronized multiplayer data collection and coordinated generation across player views.

Solaris reported a dataset containing 12.64 million multiplayer frames. It also established evaluation categories covering movement, memory, grounding, building, and view consistency.

WorldWeaver adopts that setting and retrains Solaris on the same training data for comparison. It then changes how information persists during generation.

Solaris synchronizes observations through cross-player attention over visual tokens. WorldWeaver preserves that interaction while introducing a distinct shared-state channel. The question is whether explicit state outlasts visual context more reliably.

The paper’s results favor the stateful route in this experiment. However, they do not establish that a token register always beats longer context, external memory, or a structured map.

They show something narrower and still valuable. When two video streams must describe one changing environment, explicit state supervision can improve measurable logical consistency.

World State Registers Change the Generation Mechanism

WorldWeaver separates visual synthesis from state maintenance, then trains both pathways to survive their own accumulated errors.

The model follows a three-stage training process. The first stage adapts a pretrained single-player video model into a synchronized multi-player teacher. Bidirectional attention lets that teacher observe the complete clip while learning cross-agent scene structure.

The second stage converts the teacher into a causal generator. Causal generation means each output depends only on information available at that point in the sequence. This is necessary for an interactive stream that cannot inspect future frames.

During this stage, frame tokens use recent visual context and the latest register. Register tokens read the local frame window and the preceding register. The model alternates between predicting observations and updating state.

The register receives auxiliary supervision at every committed step. A regression head predicts agent statistics. Another decoder predicts features corresponding to an aligned bird’s-eye view.

A text head predicts the scene description. The overall training objective combines these state losses with the diffusion model’s frame-generation loss.

That combined objective creates a potential conflict. Pixel generation rewards detailed visual reconstruction, while state modeling rewards compact and stable information. A single set of transformer weights must otherwise satisfy both roles.

WorldWeaver addresses that conflict with a Mixture-of-Transformers design. This architecture gives register tokens and frame tokens separate parameter branches while preserving joint attention between them.

The separation is not the same as using independent models. Visual and state tokens still exchange information inside the same interleaved sequence. However, each token type receives weights tailored to its role.

That design becomes more helpful as register supervision grows richer. The paper reports that dense shared weights can struggle when the same pathway must produce pixels and encode verifiable world semantics.

The third stage addresses another source of failure. Models commonly train on correct historical frames but deploy on their own imperfect outputs. Small errors then compound because every generated frame becomes context for the next one.

WorldWeaver adapts Self Forcing, a training method that rolls a model forward on its own generations. The technique exposes the system to its deployment conditions before inference.

Here, the problem includes both frame drift and register drift. An incorrect generated view can corrupt the next state update. A corrupted state can then distort later views for every agent.

The model therefore rolls out both frames and committed registers during training. It learns to continue from the history it actually produced, not only from clean reference data.

The inference schedule uses four denoising steps. After each frame is denoised, the model adds it to the rolling visual cache. It then updates the register before beginning the next generation step.

This cycle is the paper’s main mechanism. WorldWeaver is not merely retrieving old observations or attaching metadata after generation. State updating participates directly in the autoregressive loop.

Other research has approached the same problem with more explicit external memory. MultiGen, for example, separates memory, observation, and dynamics for editable multiplayer diffusion worlds.

MultiGen’s external representation lets users alter environmental structure. WorldWeaver keeps its state inside the generator as learned tokens. That makes its memory compact, but less directly editable.

The contrast reveals an important design choice. A structured external map provides inspectable control but requires the system to maintain that representation. An internal register offers flexibility but can hide ambiguous or incorrect state.

WorldWeaver tries to occupy the middle ground. Its registers remain latent during inference, yet explicit training targets make their contents partially recoverable. The system learns an internal state without abandoning verification entirely.

This mechanism also suggests uses beyond synthetic games. A robot fleet simulator could generate separate observations while preserving shared object locations. Training environments could model how one agent’s action changes what others encounter later.

Digital-twin systems face a related problem when different sensors provide incomplete views. Persistent state can integrate evidence across those views while the simulated environment continues evolving.

However, WorldWeaver does not test those applications. Its evidence comes from controlled Minecraft sessions with two agents, synchronized actions, and unusually accessible ground truth.

The architecture makes a clear technical claim. The broader application claim remains open.

The Scores Improve, but the Real-World Data Gap Remains

The reported gains support explicit state modeling, yet they do not show that WorldWeaver can recover complex state without simulator supervision.

The researchers evaluate movement, grounding, memory, building, and consistency. They use visual-language-model accuracy and Fréchet Inception Distance, or FID, which compares generated and reference visual distributions.

They also combine those measurements into an aggregate world score. WorldWeaver reaches 105.1 on that measure. The retrained Solaris model reaches 81.0, while a frame-concatenation baseline reaches 49.1.

The largest accuracy improvements appear in state-sensitive categories. Grounding rises from 81.3 with Solaris to 93.8 with WorldWeaver. Building increases from 9.4 to 28.1.

Consistency moves from 57.8 to 76.6. Movement also improves, reaching 82.8 compared with 79.7 for Solaris.

Memory shows a smaller advantage. WorldWeaver records 46.9, while Solaris reaches 43.8. The frame-concatenation baseline scores 37.5.

The FID results are more mixed. WorldWeaver improves several categories, including movement and consistency, but does not lead every visual comparison. Its memory FID is 64.8, compared with 61.2 for Solaris.

That split matters because logical state and visual quality are related but not identical. A model can preserve the right event while rendering it poorly. It can also create attractive frames that contradict earlier actions.

WorldWeaver’s stronger aggregate result suggests that the register helps with the combined objective. It does not eliminate every quality tradeoff.

Ablation experiments add support for the state mechanism. The researchers vary supervision types and compare shared dense weights with the separated transformer architecture.

The results indicate that different state targets help different behaviors. No single signal dominates every category. Combined supervision produces the strongest reported overall configuration.

The semi-supervised experiment addresses the most obvious scaling concern. The team fixes the labeled set at 1,000 clips, then adds increasing quantities of unlabeled video.

With no unlabeled clips, the aggregate world score is 63.2. Adding 5,000 unlabeled clips raises it to 82.3. With 10,000 unlabeled clips, the score reaches 90.3.

This result suggests that a smaller labeled collection can anchor register semantics while ordinary video improves generation. It does not remove the need for labeled state, but it reduces the required proportion in this controlled setting.

Several cautions remain.

First, the paper is an arXiv preprint and has not yet passed peer review. Its evaluation framework, aggregate score, and architectural conclusions need independent reproduction.

Second, the model operates in Minecraft. The environment has discrete blocks, accessible controller inputs, clean synchronization, and exact simulator state. Real scenes rarely offer equivalent annotations.

A warehouse robot might observe reflective surfaces, deformable objects, people, and moving equipment. There may be no reliable bird’s-eye view or complete record of hidden interactions.

Third, the experiments cover two agents. Adding more viewpoints increases both information and conflict. Registers must preserve additional identities, observations, and interactions without collapsing them into an ambiguous summary.

Fourth, the register itself can drift. Self-forcing exposes the model to its errors, but it cannot guarantee recovery after a false update. One mistaken state can influence all subsequent agents.

Fifth, the reported state is only partially interpretable. Training heads decode selected targets, but those probes do not reveal everything stored in the register. High probe accuracy does not guarantee a complete or causally correct representation.

The model also depends on automated scene descriptions. Those captions use ground-truth controller inputs alongside visual observations. Real-world data would require weaker, noisier, or estimated action labels.

The authors acknowledge the central limitation in their paper. Their main improvement depends on additional state supervision, while real-world state is more complex and often unavailable.

That admission defines the work’s actual frontier. WorldWeaver presents a credible state architecture under strong observability. It has not solved state discovery in environments where the truth is hidden.

What Will Show Whether Shared State Scales

The next evidence must test state registers beyond two-player Minecraft, isolate their computational tradeoffs, and measure recovery after incorrect updates.

The first signal is independent reproduction on the released code and evaluation setup. Researchers should verify the reported 105.1 world score and examine whether the category gains survive different random seeds.

Reproduction should also inspect the aggregate metric. A combined score can clarify broad performance, but it can obscure tradeoffs between logical accuracy and visual fidelity.

The second signal is a broader environment with incomplete supervision. A useful follow-up would retain synchronized agents while removing exact overhead maps or simulator coordinates.

That test would reveal whether registers can learn stable state from estimated geometry, noisy language, and partial action records. Success would strengthen the case for robotics and embodied simulation.

Failure would suggest that WorldWeaver’s gains depend more on privileged annotations than on the register design itself. That outcome would still inform future systems, but it would narrow the architecture’s practical reach.

The third signal is scaling across agents and horizons. Two views provide an important starting point, yet shared state becomes harder as additional actors modify the same environment.

Future evaluations should track how consistency changes with four or more agents. They should also report the point where register capacity, cache size, or accumulated error becomes limiting.

Longer rollouts need targeted stress tests. An agent could hide an object, leave the area, and return after its original frames have exited the cache. Another agent could modify that object while the first remains absent.

Those tests would separate persistent state from short-term visual recall. They would also expose whether the register updates unseen relationships or merely stores a compressed recent history.

Researchers should measure correction behavior too. If one view produces a false object or location, later evidence should repair the shared state. Otherwise, explicit memory can turn a local hallucination into a system-wide commitment.

External memory systems offer a useful comparison. Structured maps can enforce geometry and support direct editing, but they carry their own reconstruction burden. Learned registers remain flexible but harder to audit.

A convincing benchmark should compare both approaches under identical data and compute. It should include long context, latent registers, and explicit external state instead of treating one baseline as definitive.

Computational reporting will also matter. The training-only supervision heads add no inference work, according to the paper. However, register tokens and separated transformer weights still affect memory, training cost, and generation throughput.

Real-time interactive systems must balance those costs against consistency gains. A shared state that works too slowly cannot support responsive agents, games, or robotics simulations.

WorldWeaver provides a useful answer to one narrow question: recent video frames are not an adequate definition of a shared world. Its registers give the generator somewhere explicit to maintain facts across views.

The harder question is whether those facts can be learned when no simulator reveals the answer. Developers evaluating multi-agent world models should watch that boundary closely.

Do future systems keep state after privileged labels disappear? Can they correct a shared memory after one agent introduces an error? Do they remain coherent as the number of agents grows?

Those tests will determine whether WorldWeaver marks a reusable architecture or a strong Minecraft result. For now, it makes the debate more precise: a world model must remember the world, not only its latest pictures.