🤖 AI

ReWorld: An Interactive World Model with Long-Horizon Memory

ReWorld is an interactive world model that resolves the structural tension between short-horizon control and long-horizon memory by separating these capabilities during training with mixed attention mechanisms and unifying them at inference via a pose-indexed landmark retrieval system, achieving superior action fidelity, video quality, and minute-long temporal consistency across diverse visual domains.

Original authors: Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang, Tianyi Gui, Lianghua Huang, Yingcong Chen

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang, Tianyi Gui, Lianghua Huang, Yingcong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer that can watch a video and then predict what happens next, frame by frame, as if it were living inside the scene. This is the promise of a "world model," a type of artificial intelligence designed to simulate environments where an agent can move and act. For these systems to feel truly alive, they must do three difficult things at once: they must react instantly to a user's commands, they must remember exactly what a place looked like when the user left and returned to it, and they must keep generating new video in real time without running out of memory. The challenge has been that these goals often fight each other. To react quickly, a system needs to focus only on the immediate past, but to remember a distant location, it needs to hold onto a vast history. If a system tries to do both with the same simple approach, it usually fails at one or the other, either forgetting the past or stumbling over the present.

Researchers have now built a system called ReWorld that solves this tension by teaching the computer to think in two different ways simultaneously. The team, working with institutions in Hong Kong and Alibaba, created a model that can follow a user's camera movements, stream high-quality video of photorealistic and game-like worlds, and recall specific scenes from minutes ago, even after the camera has traveled far away and come back. The key discovery is that the system does not need to choose between short-term reaction and long-term memory; instead, it separates these tasks during its learning process and then manages them carefully when it is actually running.

The core of ReWorld's success lies in how it organizes its attention. In many artificial intelligence models, the system looks at all the information it has seen so far to decide what to generate next. ReWorld splits this job among different parts of its brain. Some parts are trained to look only at the very recent past, perhaps the last few seconds, to ensure that a button press results in an immediate and accurate camera movement. Other parts are trained to look at the entire history of the video, allowing the system to recognize that a specific rock formation or building it saw a long time ago is the same one it is seeing again now. Crucially, the researchers did not assign these jobs to fixed parts of the system. Instead, they shuffled the roles randomly during training, ensuring that every part of the system learned how to handle both immediate reactions and distant memories. This prevents the system from becoming dependent on a specific way of looking at the past, which is vital because, in the real world, the system cannot keep its entire history in memory forever.

When the system is put to work, it faces a strict limit on how much information it can hold at any one moment. To handle this, ReWorld uses a clever filing system. As the video plays, it keeps a small, constantly updated window of the most recent frames. When older frames fall out of this window, they are not simply deleted. Instead, the system checks if the camera has moved far enough to warrant saving a snapshot of that location. If it has, the system saves a high-quality "landmark" of that scene. These landmarks are stored in a separate, limited bank. When the camera turns around and heads back toward a place it visited earlier, the system searches this bank for the landmark that matches the current view and pulls it back into the active memory. This allows the system to remember a scene from a minute ago without needing to store every single frame that happened in between.

To make this work, the researchers had to teach the model using a very specific kind of data. They combined footage from real-world videos, video games, and computer-generated environments, but they did something unusual: they made sure that a single command, like pressing a key to move forward, moved the camera the exact same physical distance in every single type of footage. This created a unified language of movement. They also used a special training technique where they would sometimes hide parts of the video history from the model, forcing it to learn how to reconstruct the scene even when its memory was incomplete. This prepared the model for the reality of its limited memory bank, ensuring it wouldn't get confused when it had to rely on a few key landmarks rather than a continuous stream of data.

The results of this approach are striking. In tests where the camera moved away and then returned to its starting point, ReWorld was able to regenerate the original view with remarkable accuracy, even after the camera had traveled for a minute or more. Other systems, which rely on keeping a simple sliding window of recent frames, failed to remember the starting point once it fell out of that window. ReWorld also proved superior at following commands, with the camera moving exactly as intended without drifting off course. The system can generate video at a resolution of 704 by 1280 pixels, fast enough to feel interactive, and it maintains this quality across different styles, from realistic landscapes to stylized game worlds.

The researchers found that their method works because it respects the different needs of control and memory. By training the system to handle short-term commands and long-term recall separately, and then using a smart retrieval system to bring back old memories only when needed, they created a world model that feels both responsive and consistent. The system does not need to be a massive, all-knowing entity to remember the past; it just needs to know how to find the right piece of the past when it is needed. This approach allows the model to run on standard computer hardware while maintaining a sense of continuity that was previously impossible for interactive video generation. The work suggests that the future of interactive AI lies not just in making models bigger, but in making them smarter about how they organize and access the information they have already learned.

Technical Summary: ReWorld – An Interactive World Model with Long-Horizon Memory

1. Problem Statement

Interactive world models aim to simulate environments where an agent can act, generating coherent, explorable video streams in response to user actions. However, current systems face a fundamental structural tension between control and memory:

Control requires short attention horizons to react immediately to user inputs (e.g., key presses) without being distracted by distant history.
Memory requires unbounded attention horizons to recall and accurately regenerate scenes visited long ago.

Existing approaches often struggle to balance these. Training both capabilities simultaneously typically leads to a trade-off where improving control degrades memory (revisit fidelity), and vice versa. Furthermore, deploying these models in real-time is constrained by the KV cache memory budget; maintaining a full history for long rollouts is computationally infeasible, while standard sliding windows inevitably evict the evidence needed for long-horizon recall.

2. Methodology

ReWorld addresses these challenges through a two-step design: decoupling the training of control and memory via window splitting, and consolidating memory under a fixed inference budget.

2.1 Training: Window-Split Control and Memory

The core innovation is separating the learning of control and memory by assigning different attention windows to different attention heads within the same model iteration.

Mixed Per-Head Attention Windows: The model splits its attention heads into two groups:
- Local Heads (18/24): Attend only to the most recent $w=12$ frames (3 chunks). These heads learn control, ensuring the model responds correctly to immediate actions regardless of the total history length.
- Global Heads (6/24): Attend to the entire causal history. These heads learn memory, enabling the retrieval of distant content.
Random Head Routing: To prevent specific heads from becoming permanently specialized (which would fail at inference when the full history is unavailable), the assignment of global vs. local roles is randomized at every optimizer step. A fixed pool of random partitions is cycled through, ensuring every head learns both capabilities. This allows the model to function correctly even when the inference cache is bounded and shared by all heads.
Dual Conditioning: The model utilizes two conditioning channels:
1. Pose-Indexed Attention (MRoPE): Embeds camera pose ( $SE(3)$ ) directly into the attention mechanism (via phase offsets and residuals), allowing the model to retrieve content based on spatial proximity rather than just temporal distance.
2. Direct Action Injection: Explicitly injects 6-DoF camera commands (as Plücker ray maps) to ensure precise control authority.

2.2 Inference: Memory Consolidation and Chunk-Drop Training

At inference, the model operates under a strict bounded KV cache budget ( $B=12$ chunks).

Memory Consolidation: Instead of a sliding window that discards old data, ReWorld maintains a bounded landmark bank ( $K=30$ $K = 30$ ).
- Admission: As chunks age out of the recent window, they are admitted to the landmark bank only if the camera has traveled a significant distance since the last stored landmark (an "odometer rule"). This ensures the bank contains diverse, non-redundant snapshots of the environment.
- Eviction: When the bank is full, the most spatially redundant member (closest in pose space to others) is evicted.
- Retrieval: The fixed cache is populated with a "sink" chunk (for global layout), the 5 most recent chunks, and the 6 landmarks ( $|T_k|=6$ ) closest to the current camera pose, totaling the budget of $B=12$ .
Chunk-Drop Training: To bridge the gap between training (contiguous history) and inference (sparse, non-contiguous history), the training process randomly masks a subset of past KV chunks. This teaches the model to reconstruct scene states from incomplete memory, making the sparse caches produced by the consolidation policy "in-distribution."

2.3 Real-Time Distillation

To achieve real-time performance, ReWorld employs Distribution-Matching Distillation (DMD) confined to a lightweight LoRA adapter.

A multi-step teacher model trains a student model that operates in just 4 denoising steps.
The LoRA adapter is applied only to the attention and feed-forward layers, leaving the action injection and pose conditioning paths untouched. This allows a single backbone to support both high-fidelity multi-step generation and real-time interactive streaming.

2.4 Data Pipeline

The model is trained on a metric-scale-aligned corpus of 220,724 clips from eight sources (Unreal Engine, real-world footage, and game data).

Scale Alignment: All sources are rescaled so that a single physical action (e.g., a key press) corresponds to the same physical distance across all datasets.
Palindrome Augmentation: Trajectories that retrace their path (out-and-back) are used to provide explicit supervision for long-horizon memory.

3. Key Contributions

Window-Split Training: A novel training recipe using mixed per-head attention windows and random head routing to decouple control (short-window) and memory (long-window) learning without increasing model parameters.
Consolidation-Based Inference: A memory management system that uses a pose-indexed landmark bank with redundancy-based eviction, paired with chunk-drop training, to maintain spatial memory over unbounded rollouts within a fixed KV budget.
Metric-Aligned Multi-Source Pipeline: A unified data engine that aligns synthetic, real, and game footage to a single physical action scale, enabling robust transfer across domains.
Real-Time Deployment: A LoRA-confined distillation strategy that enables a single backbone to stream high-resolution (704×1280) video in real-time (4 steps) while maintaining high-fidelity capabilities.

4. Results

Evaluated against six recent interactive world models (including SANA-WM, DreamX, HY-WorldPlay, and Matrix-Game 3.0), ReWorld demonstrates superior performance across three axes:

Control Fidelity: ReWorld achieves the best rotation error (11.95°) and camera motion consistency on a benchmark of 240 clips across six canonical trajectories. It outperforms baselines in both translation-heavy and rotation-heavy tasks.
Long-Horizon Memory: On "needle-in-a-haystack" palindrome tests (up to 384 latents / ~64 seconds), ReWorld maintains high revisit fidelity.
- At a rollout length of 96 latents, where sliding windows have evicted the starting view, ReWorld achieves a revisit SSIM of 0.384, significantly outperforming baselines that drift or alter geometry.
- Ablation studies confirm that the random head routing is critical: without it, adding action injection degrades memory; with it, control and memory are preserved simultaneously.
Generation Quality: ReWorld attains the highest mean score on VBench (0.850), balancing temporal consistency, motion smoothness, and dynamic degree. Unlike some baselines that sacrifice motion for consistency, ReWorld maintains high dynamic degrees.

5. Significance and Claims

The paper claims that ReWorld resolves the structural tension between control and memory by separating them during training and bounding them at inference.

Decoupling: The random head routing mechanism proves that control does not need to depend on specific heads or long windows, allowing the model to function correctly even when the inference cache is compressed.
Scalability: By consolidating memory into a pose-indexed landmark bank, the model can generate minute-long rollouts with consistent spatial memory at a constant computational cost, a feat where sliding windows fail and full-KV attention runs out of memory.
Practicality: The system demonstrates that high-fidelity, real-time interactive world models are achievable through efficient distillation and careful architectural design, rather than simply scaling up model size or context length.

The authors note that while the current system excels at navigational interaction, extending this consolidation mechanism to dynamic scenes and richer, non-navigational interactions remains a future direction.