🤖 AI

Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory

The paper introduces Infinite-World, a robust interactive world model that achieves coherent 1000+ frame visual generation in real-world environments by employing a hierarchical pose-free memory compressor, an uncertainty-aware action labeling module, and a revisit-dense finetuning strategy to overcome challenges posed by noisy pose estimations and scarce viewpoint revisits.

Original authors: Ruiqi Wu, Xuanhua He, Meng Cheng, Tianyu Yang, Yong Zhang, Zhuoliang Kang, Xunliang Cai, Xiaoming Wei, Chunle Guo, Chongyi Li, Ming-Ming Cheng

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruiqi Wu, Xuanhua He, Meng Cheng, Tianyu Yang, Yong Zhang, Zhuoliang Kang, Xunliang Cai, Xiaoming Wei, Chunle Guo, Chongyi Li, Ming-Ming Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a video game where you can walk around a room, look out a window, and turn corners. Most "world models" (AI systems that try to understand and predict how a world looks as you move) are like short-term memory devices. They can show you what happens for a few seconds, but if you try to walk around for a long time—say, 1,000 steps—they start to get confused. They might forget what the door looked like, or they might suddenly make the walls disappear.

The paper "Infinite-World" introduces a new AI system designed to solve this problem. It can simulate a world for over 1,000 frames (roughly 30–40 seconds of continuous video) without losing its mind or forgetting the layout of the room.

Here is how they did it, explained through simple analogies:

1. The Problem: The "Blurry Map" and the "Jittery Camera"

Real-world videos are messy.

The Jitter: When you film a video with a handheld camera, your hand shakes. The AI doesn't know if you moved the camera or if the camera just shook. This makes it hard to teach the AI how to move the world accurately.
The Memory Limit: Most AIs try to remember every single frame they have ever seen. If you ask them to remember 1,000 frames, their "brain" gets too full, and they crash or start hallucinating (making up things that aren't there).

2. The Solution: Three Smart Tricks

Trick A: The "Summarizing Librarian" (Hierarchical Pose-Free Memory Compressor)

Imagine you are reading a very long book. If you try to remember every single word, you will forget the beginning by the time you reach the end.

What other AIs do: They try to keep the whole book open on the table. It gets messy and heavy.
What Infinite-World does: It uses a "Summarizing Librarian."
- Short term: For the last few pages, the librarian keeps the details sharp.
- Long term: As you go further back in the book, the librarian stops remembering every word. Instead, they write a summary of the previous chapters.
- The Magic: This summary is compressed into a fixed size. No matter if you read 10 pages or 1,000 pages, the librarian only needs to keep one small summary card in their pocket. This allows the AI to remember the "gist" of the room (where the window is, where the desk is) without getting overwhelmed by data. It does this without needing a GPS or a camera map (pose-free); it just learns to summarize what matters.

Trick B: The "Traffic Light" for Movement (Uncertainty-Aware Action Labeling)

Real-world video data is noisy. Sometimes the camera moves because you walked; other times, it moves because your hand shook.

The Problem: If you teach an AI "Move Left" based on a shaky video, it might learn that "Move Left" actually means "Shake the camera."
The Solution: The authors created a Traffic Light System for movement:
- Green (Go): The movement is clear and strong. The AI learns this as a real action.
- Red (Stop): The movement is tiny or just noise. The AI ignores it.
- Yellow (Uncertain): The movement is blurry or confusing. Instead of forcing the AI to guess, the system labels it "Uncertain" and tells the AI, "Don't learn from this specific moment."
- Result: The AI learns only from the clear, strong movements, making it much better at responding to your controls without getting confused by camera shakes.

Trick C: The "Practice Run" (Revisit-Dense Finetuning)

The authors realized that to teach an AI to remember a long path, you don't need a million hours of random video. You need a specific kind of practice.

The Insight: If you walk in a straight line for 10 miles, you never see where you started. But if you walk in a circle and come back to the start, you learn the shape of the world.
The Strategy: They took a tiny dataset (only 30 minutes long) where the camera went in circles and revisited the same spots over and over. They used this to "wake up" the AI's long-term memory. It's like giving a student a quick, focused quiz on a specific concept to make sure they really understand it, rather than making them read a whole library of random books.

The Result

When you put these three tricks together, Infinite-World becomes a master storyteller.

It can watch you walk around a room for a long time.
It remembers that the window was on the left, even after you've walked past it and come back around.
It responds to your "move left" commands accurately, even if the training video was a bit shaky.
It does all this without needing a super-computer to hold all the data in its memory at once.

In short, they built an AI that can play "explore the world" for a long time without getting lost, forgetting, or crashing, using smart summarization and careful filtering of messy data.

Technical Summary: Infinite-World

Problem Statement

Existing interactive world models have achieved remarkable success in synthetic environments (e.g., Unreal Engine 5, GTA5) where perfect ground-truth camera poses and paired video-action data are available. However, transitioning these models to real-world scenarios faces three critical barriers:

Inaccurate Pose and Unreliable Control: Real-world video requires estimated camera extrinsics, which introduce noise and errors, degrading the precision of action-response mapping.
Scarcity of Viewpoint Revisits: Natural video streams are predominantly linear, lacking the dense "loop-closure" (returning to previous locations) necessary for models to learn global geometric structures and long-range spatial memory.
Computational Bottlenecks: Standard attention mechanisms scale quadratically ( $O(L^2)$ ) with sequence length, making 1000+ frame generation computationally prohibitive. Existing memory solutions either rely on error-prone pose metadata for retrieval or use simple downsampling that causes severe information loss.

Methodology

The authors propose Infinite-World, a robust interactive world model designed to maintain coherent visual memory over 1000+ frames using raw, uncurated real-world video. The framework consists of three core components:

1. Hierarchical Pose-free Memory Compressor (HPMC)

To address the computational bottleneck and eliminate the need for explicit geometric priors, the authors introduce a two-stage recursive distillation pipeline:

Local Compression: For short-range sequences, a temporal encoder compresses raw latents directly, reducing temporal resolution by a factor of $k=4$ .
Global Compression: For long-horizon sequences exceeding a memory budget, the model employs an adaptive sliding-window mechanism. Raw latents are partitioned into overlapping chunks, locally compressed, and then recursively distilled into a unified global representation.
Joint Optimization: The compressor is trained end-to-end with the DiT (Diffusion Transformer) backbone. This allows the model to autonomously learn which historical cues are salient for future synthesis and loop-closure without external pose metadata, maintaining a strictly bounded memory footprint ( $T_{max}$ ) regardless of the total sequence length.

2. Uncertainty-aware Action Labeling

To handle noisy pose estimations in real-world data, the authors propose a strategy that discretizes continuous motion into a tri-state logic:

No-operation: Assigned when motion magnitude is below a noise floor threshold ( $\tau_1$ ).
Discrete Action: Assigned when motion exceeds a trigger threshold ( $\tau_2$ ), mapped to semantic directions (e.g., W/A/S/D for translation).
Uncertain: Assigned to ambiguous, low signal-to-noise ratio (SNR) motions between the two thresholds.
Instead of discarding these ambiguous samples, the model explicitly labels them as "Uncertain." This shields the deterministic action space from being corrupted by jitter or slow-drifting poses while maximizing the utilization of raw video data.

3. Revisit-Dense Finetuning Strategy

Guided by a pilot study revealing that spatial memory activation depends more on the topological density of trajectories than total data volume, the authors employ a two-stage training approach:

Pre-training: The model is trained on a large-scale, open-domain dataset (30+ hours) to learn diverse visual priors and local dynamics.
Finetuning: A compact, 30-minute Revisit-Dense Dataset (RDD) is used to activate long-range consistency. This dataset features frequent viewpoint revisits and loop-closures, enabling the model to learn global spatial structures efficiently without requiring massive data quantities.

Key Contributions

Pose-Free Long-Horizon Modeling: The introduction of the HPMC enables interactive world modeling to scale to 1000+ frames with bounded computational cost, eliminating the dependency on explicit camera poses.
Robust Training Paradigm: A novel framework for real-world raw videos featuring uncertainty-aware action labeling to mitigate pose noise and a revisit-dense finetuning strategy to activate long-range memory.
Empirical Validation: Extensive experiments demonstrate that the model achieves superior performance in visual quality, action controllability, and spatial consistency compared to state-of-the-art baselines.

Results

The authors evaluated Infinite-World against several baselines, including HY-World 1.5, Hunyuan-GameCraft, Matrix-Game 2.0, and Yume 1.5, using a benchmark of 100 diverse scenarios with 16-chunk (approx. 1000-frame) trajectories.

Quantitative Metrics: Infinite-World achieved top or near-top scores across VBench metrics (Motion Smoothness, Dynamic Degree, Aesthetic Quality).
User Study: In a large-scale user study with ELO ratings, Infinite-World achieved a dominant score of 1719, significantly outperforming the next best model (HY-World 1.5 at 1542).
- Memory Consistency: Ranked 1st (1.92), demonstrating effective mitigation of error accumulation over long horizons.
- Visual Fidelity: Ranked 1st (1.67), preserving global landmarks and structural details.
- Action Responsiveness: Ranked 1st (1.54), proving that the model can learn robust action-response mappings even from noisy, imperfect real-world trajectories.
Qualitative Analysis: Visual comparisons show that while baselines suffer from ghosting, structural distortion, or a "forward-moving" bias, Infinite-World successfully performs long-range loop-closures and maintains scene coherence after hundreds of frames.
Efficiency: Memory consumption analysis shows that while uncompressed baselines fail at ~180 frames due to OOM errors, Infinite-World stabilizes at ~45GB, supporting inference beyond 1300 frames.

Significance and Claims

The paper claims that Infinite-World successfully bridges the "reality gap" in world modeling. By removing the reliance on perfect geometric priors and developing a mechanism to handle noisy control signals, it demonstrates that stable, long-horizon interactive simulation is achievable in complex real-world environments.

The authors emphasize that their work provides an open and efficient framework for long-horizon real-world interactive modeling, contrasting with closed-source state-of-the-art systems. They conclude that the bottleneck for long-horizon modeling lies not in data quantity, but in the topological density and duration of individual trajectories, a finding validated by their ability to activate 1000-frame consistency with only 30 minutes of finetuning data. The work aims to serve as a foundation for future real-world neural simulators in domains such as autonomous driving and embodied AI.