🤖 AI

WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models

This paper introduces WorldRoamBench, a novel open-world benchmark designed to rigorously evaluate the long-horizon stability of interactive world models across four critical dimensions—action, vision, physics, and memory—revealing that current state-of-the-art models still struggle to achieve reliable, physically grounded, and memory-faithful performance in continuous interactive environments.

Original authors: Ting-Bing Xu, Jiacheng Sui, Zhe Gao, Kewei Shi, Wenjin Yang, Zhicheng Liu, Zhaoxu Sun, Mingchao Sun, Hongyu Pan, Fan Jiang, Mu Xu, Qi Fan, Yong Li, Baoquan Chen

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ting-Bing Xu, Jiacheng Sui, Zhe Gao, Kewei Shi, Wenjin Yang, Zhicheng Liu, Zhaoxu Sun, Mingchao Sun, Hongyu Pan, Fan Jiang, Mu Xu, Qi Fan, Yong Li, Baoquan Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a video game where you can press keys on your keyboard (W, A, S, D) to walk, turn, and look around in a world that doesn't exist yet. Instead of a pre-made map, an AI generates the scenery in real-time as you move. This is called an Interactive World Model.

The paper introduces a new "report card" called WorldRoamBench to test how good these AI worlds really are when you play for a long time (10 to 60 seconds), rather than just a few seconds.

Here is how the paper breaks down the testing, using simple analogies:

1. The Problem: The "Short Clip" Trap

Previous tests only watched the AI for a few seconds. It's like judging a marathon runner by watching them tie their shoes. The AI might look perfect for 5 seconds, but then it starts to glitch, forget where it was, or walk through walls. WorldRoamBench forces the AI to run the whole marathon to see if it falls apart.

2. The Four Tests (The "Report Card")

The benchmark checks the AI on four specific skills:

A. Action Following: "The Obedient Pet"

The Test: You press "Forward." Does the AI actually move forward?
The Old Way: Previous tests looked at the overall path. If you pressed "Forward" but the AI zig-zagged wildly before ending up in the right spot, it got a good score.
The New Way: WorldRoamBench checks every single step. It's like checking if a dog sits every time you say "Sit," not just if it ends up in the corner. It reveals if the AI is ignoring your keys or getting confused at the moment you change direction.

B. Visual Quality: "The Fading Photograph"

The Test: Does the world stay clear and beautiful, or does it turn into a blurry mess?
The Old Way: They averaged the quality. If the start was pretty and the end was ugly, the average might still look okay.
The New Way: They look for the "Mid-Sequence Collapse." Imagine a photo that starts sharp, gets blurry in the middle, and then somehow gets sharp again at the end. The old tests would miss the blurry middle. This test catches that "glitchy dip" where the AI loses its mind for a moment.

C. Interaction Physics: "The Reality Check"

The Test: Does the world obey the laws of physics?
The Old Way: They mostly ignored this or checked it very loosely.
The New Way: They test three specific things:
- Mechanics: If you walk into a wall, do you stop? Or do you walk through it like a ghost? If you drop a cup, does it fall?
- Optics: Do shadows move correctly as you turn? Do reflections in a mirror look right?
- 3D Consistency: If you walk down a long hallway, do the walls stay straight, or do they warp and twist like a funhouse?

D. Memory: "The Time Traveler"

The Test: If you walk away from a tree and then walk back, is it the same tree?
The Old Way: They compared the first frame and the last frame. But if the AI walked slightly off-course, the frames wouldn't match, and they blamed the AI's memory, even if the AI just walked in a slightly different circle.
The New Way: They use a "3D Point Cloud" (a digital map of the room). They check if the shape of the room is preserved, regardless of exactly where the camera is standing.
- Scene Memory: Is the building I saw earlier still there?
- Subject Memory (for 3rd person): If I'm watching a character, does that character look like the same person, or do they turn into a blob or a different animal?

3. The Results: "No Perfect Player"

The authors tested over 10 different AI models (some open-source, some from big tech companies like Google and Alibaba).

The Big Finding: No single model is perfect at everything.
Some models are great at following your commands but terrible at physics (they walk through walls).
Some are beautiful to look at but forget what they generated 10 seconds ago.
Some are very good at physics but get confused when you ask them to turn quickly.

4. Why This Matters

The paper argues that to build a truly immersive, infinite world (like the "Metaverse" or advanced video games), we can't just have pretty pictures. The world needs to be stable (not glitching), physical (obeying gravity), and memorable (remembering what you saw).

WorldRoamBench is the first tool that rigorously checks all these things at once, showing us exactly where current AI is failing so developers know what to fix next. It's not about making the AI "smarter" in a general sense, but making it a more reliable, consistent, and physically grounded virtual world.

Technical Summary: WorldRoamBench

Problem Statement

Despite rapid advancements in Interactive World Models (IWMs) capable of generating photorealistic environments in response to user inputs (e.g., WASD keys), existing benchmarks fail to adequately evaluate long-horizon stability. Current evaluation methods suffer from four critical limitations:

Action Evaluation: Most benchmarks rely on trajectory-level metrics (e.g., Relative Pose Error, Absolute Trajectory Error) derived from recovered 3D camera paths. These metrics suffer from cross-model semantic scale disparities (where different models interpret the same action with vastly different magnitudes) and can mask per-frame failures (e.g., a model ignoring a keystroke and then over-compensating).
Visual Stability: Existing metrics often measure average visual quality, ignoring temporal degradation. Autoregressive models frequently exhibit non-monotonic quality collapse (drift, blurring, structural breakdown) mid-sequence that start-vs-end comparisons miss.
Interaction Physics: Current benchmarks lack systematic evaluation of physical plausibility under active control, such as collision response, clipping, gravity, and optical consistency (reflections/shadows).
Memory Consistency: Memory is often evaluated via symmetric frame-pair comparisons assuming perfect action execution. In reality, cumulative drift causes the "revisit" frame to spatially offset from the ground truth, conflating action-following errors with memory degradation.

Methodology

The authors introduce WorldRoamBench, an open-world benchmark designed to evaluate IWMs across 10–60 seconds of continuous interaction in diverse scenarios (Nature, Urban, Indoor) and perspectives (First-Person View - FPV, Third-Person View - TPV). The benchmark comprises 600+ test cases and evaluates 10+ state-of-the-art models (both open-source and closed-source).

The evaluation framework is built on four complementary dimensions:

1. Action Following

Per-Frame Discrete Action Accuracy: Instead of recovering trajectories, the benchmark discretizes the estimated camera pose (using ViPE) into atomic actions (W, A, S, D, I, J, K, L) using a latent-stride mechanism (grouping frames into windows of $K=4$ to match model generation cadence). It reports both Strict Accuracy (exact match of all sub-actions) and Partial Accuracy (match of at least one sub-action).
Trajectory Score (TrajScore): A geometric metric that compares the generated path against an adaptive ground truth. The adaptive GT preserves the model's own displacement magnitude but imposes the ideal direction and shape (e.g., a straight line for "forward"). This decouples "how far" the model moved from "whether it moved correctly." Errors are normalized by path length and rotation to ensure cross-model comparability.

2. Visual Quality

Absolute Quality: Measured via Imaging Quality (MUSIQ) and Aesthetic Quality (CLIP-based predictor).
Temporal Drift: A segment-based metric that splits the rollout into $N=10$ windows to identify the largest quality drop ( $D = \frac{\bar{s}_{best} - \bar{s}_{worst}}{|\bar{s}_{best}|}$ ). This captures mid-sequence collapses that average scores miss.

3. Interaction Physics

A controllability-gated evaluation ensures the model responds to actions before assessing physics. It covers three domains using VLM-based (Qwen3-VL-PLUS) judgment:

Mechanics: Collision, clipping (passing through solids), deformation (footprints/ripples), terrain following, and gravity.
Optics: Reflection plausibility and shadow consistency.
3D Consistency: Preservation of rigid spatial structure and depth ordering during camera motion.

4. Memory

Memory is evaluated under imperfect control using two tracks:

Scene Memory: Instead of frame-pair comparison, the benchmark localizes the observation-revisit transition, reconstructs point clouds for both segments, and aligns them. It calculates an F1 score based on Retention (observed geometry recovered) and Hallucination (revisit geometry unsupported by observation).
Subject Memory (TPV only): Uses tracking (SAM2) and VLM reasoning to evaluate the preservation of the protagonist's identity, structure, and appearance over time.

Key Results

The benchmark evaluated 10+ models, including closed-source leaders (Genie 3, Happy Oyster) and open-source models (Matrix-Game 2.0/3.0, HY-World 1.5, Lyra 2.0, SANA-WM, etc.).

No Dominant Model: No single model satisfies all dimensions. Closed-source models generally excel in physics and memory, while open models often lead in action following and visual quality.
Trajectory $\neq$ Per-Frame Correctness: Models with high trajectory scores (>85) can exhibit low strict action accuracy (<65%), revealing hidden failures where models drift along the correct path but miss individual keystrokes.
Visual Quality $\neq$ Controllability: High aesthetic scores do not guarantee accurate action following; visual fidelity and controllability appear to be largely independent capabilities.
Physics vs. Action Trade-off: Models that strictly adhere to physics (e.g., stopping at walls) often deviate from predefined trajectories, resulting in lower action-following scores despite physically plausible behavior.
Third-Person Controllability: There is a significant gap in TPV control rates. Happy Oyster achieved an 84.43% control rate, while HY-World 1.5 struggled at 21.23%, indicating that third-person stability is not yet a solved problem for many open models.

Significance and Claims

The paper claims that WorldRoamBench provides a diagnostic foundation for measuring progress toward genuinely interactive and stable world simulations. By exposing failure modes hidden by short clips and trajectory-only metrics, the benchmark highlights that current IWMs are not yet reliable for real-world deployment.

The authors emphasize that future world models must integrate action responsiveness, physical grounding, and persistent scene/subject memory simultaneously. The benchmark serves as a standardized tool to drive research in these specific, often decoupled, areas, moving the field beyond "photorealistic video generation" toward "stable, physically grounded interactive environments."