💻 computer science

World-in-World: World Models in a Closed-Loop World

This paper introduces "World-in-World," the first open platform for benchmarking generative world models in closed-loop embodied settings, revealing that controllability, action-observation scaling, and inference-time compute are more critical for task success than visual quality alone.

Original authors: Jiahan Zhang, Muqing Jiang, Nanru Dai, Taiming Lu, Arda Uzunoglu, Shunchi Zhang, Yana Wei, Jiahao Wang, Vishal M. Patel, Paul Pu Liang, Daniel Khashabi, Cheng Peng, Rama Chellappa, Tianmin Shu, Alan Y

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiahan Zhang, Muqing Jiang, Nanru Dai, Taiming Lu, Arda Uzunoglu, Shunchi Zhang, Yana Wei, Jiahao Wang, Vishal M. Patel, Paul Pu Liang, Daniel Khashabi, Cheng Peng, Rama Chellappa, Tianmin Shu, Alan Yuille, Yilun Du, Jieneng Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to navigate a cluttered room to find a specific object. To succeed, it cannot simply react to what it sees in the immediate moment; it must anticipate what will happen if it takes a step forward, turns left, or reaches out. It needs a mental model of the world, a way to simulate the future before it actually happens. For years, scientists have been building artificial intelligence systems that can generate stunningly realistic videos of moving scenes. These systems, often called world models, can create images of a car driving down a street or a cup falling off a table with such visual fidelity that they look like real footage. However, a critical question has remained unanswered: does this visual beauty actually help a robot make better decisions? Until now, the field has been dominated by benchmarks that judge these models solely on how pretty their generated videos look, ignoring whether those videos are useful for a robot trying to complete a real task.

A team of researchers has introduced a new approach called World-in-World, which shifts the focus from visual perfection to practical utility. Instead of asking "how good does this video look?", they ask "does this video help the agent succeed?" The researchers built a closed-loop testing platform where artificial agents must perform four distinct physical tasks: finding a hidden object by actively looking around, navigating to a location based on a single reference image, answering questions about a 3D environment after exploring it, and controlling a robotic arm to move objects. In this setup, the world model acts as a simulator. Before the robot commits to a physical action, it uses the model to imagine several possible futures. It then evaluates these imagined scenarios to choose the path most likely to lead to success, effectively thinking ahead before acting.

The results of this new evaluation revealed a surprising disconnect between visual quality and task performance. The researchers tested a wide range of state-of-the-art video generators, including some that produce incredibly high-definition and aesthetically pleasing clips. They found that high visual quality alone does not guarantee that a robot will succeed at its task. A model might generate a beautiful video of a room, but if the physics in that video are slightly off or if the robot cannot precisely control the camera movement within the simulation, the robot will make poor decisions. In fact, the study showed that controllability—the ability to steer the simulation with specific, low-level actions—matters far more than the sheer beauty of the image. A model that responds reliably to a command to "move forward two steps" is more valuable than a model that produces a stunning but uncontrollable video.

Furthermore, the team discovered that simply using a more powerful pre-trained video generator is not the best way to improve a robot's performance. Instead, they found that taking a standard video generator and fine-tuning it with a relatively small amount of data specific to the robot's environment and actions yields much better results. This process, known as post-training, aligns the model's predictions with the specific rules of the physical world the robot inhabits. The data also showed a clear scaling law: the more examples of action and observation the model sees during this fine-tuning, the better it becomes at helping the robot succeed. Additionally, giving the system more time to compute—allowing it to simulate more potential futures before making a decision—significantly improved its success rate.

The study concludes that for world models to be truly useful in robotics and embodied AI, they must be judged by their ability to support decision-making in a closed loop, not by their ability to create flawless visuals. While current models still struggle with the complex physics of robotic manipulation, such as the precise friction and contact between objects, the new framework provides a clear path forward. By prioritizing controllability and adapting models to specific tasks through post-training, researchers can build systems that do not just look like the real world, but actually understand how to interact with it. This shift from judging the picture to testing the plan marks a significant step toward creating intelligent agents that can reliably navigate and manipulate the physical world.

Technical Summary: World-in-World: World Models in a Closed-Loop World

Problem Statement

Recent advances in visual generation have produced world models capable of simulating environments with striking visual realism. However, the community lacks a unified framework to evaluate whether these models genuinely enhance embodied agent performance. Existing benchmarks (e.g., VBench, WorldModelBench) primarily focus on isolated visual quality metrics or visual plausibility, often using open-loop protocols that do not test the model's utility in decision-making. Consequently, it remains unclear if high-fidelity video generation translates to successful task completion in closed-loop interactions where agents must perceive, plan, and act based on predicted futures.

Methodology

The authors introduce World-in-World, the first open platform designed to benchmark world models (WMs) within a closed-loop environment that mirrors real agent-environment interactions. The methodology comprises four core components:

1. Unified Closed-Loop Online Planning Strategy

The framework employs a policy-guided beam search strategy consisting of three iterative stages:

Proposal: An agent policy ( $\pi_{proposal}$ ) generates $M$ candidate action sequences ( $\hat{A}_t$ ) based on the current observation ( $o_t$ ) and task goal ( $g$ ).
Simulation: A Unified Action API transforms these candidate plans into control inputs ( $I_t$ ) compatible with diverse world models (e.g., text prompts, camera trajectories, or low-level action sequences). The world model ( $g_\theta$ ) then performs counterfactual rollouts to predict future states ( $\hat{O}_t$ ) for each plan.
Revision: A revision policy ( $\pi_{revision}$ ) scores the simulated trajectories and selects the optimal decision ( $D^*_t$ ) to execute in the real environment. The loop repeats with the new observation.

2. Unified Action API

To integrate heterogeneous world models, the authors propose an API that maps agent action sequences to specific model inputs:

Text Prompts: For image-to-video models, actions are converted into descriptive text.
Camera Trajectories: For viewpoint-conditioned models, actions are mapped to camera movements (translation/rotation).
Low-level Actions: For models accepting discrete/continuous actions, a direct mapping to the model's action vocabulary is applied.

3. Comprehensive Embodied Tasks

The benchmark evaluates WMs across four distinct tasks spanning perception, navigation, and manipulation:

Active Recognition (AR): Identifying occluded targets via active exploration (Habitat-Sim/Matterport3D).
Image-Goal Navigation (ImageNav): Reaching a target viewpoint given a reference image (HM3D).
Active Embodied Question Answering (A-EQA): Answering open-ended questions after active exploration (OpenEQA/HM3D).
Robotic Manipulation: Controlling a robotic arm for grasping and placement (RLBench/CoppeliaSim).

4. Post-Training Protocol

To adapt pretrained video generators for embodied tasks, the authors introduce a fine-tuning procedure using action-observation data from the target simulators (Habitat-Sim and RLBench). This aligns the generative prior with the specific domain distribution and action space, ensuring the model learns to predict outcomes conditioned on agent actions rather than just natural video statistics.

Key Contributions

World-in-World Benchmark: A comprehensive closed-loop evaluation platform that prioritizes task success over visual fidelity, evaluating WMs across four diverse embodied tasks.
Unified Framework: A standardized planning strategy and action API that allows heterogeneous world models (from image-based to video-based, and from open-source to proprietary) to be seamlessly integrated and compared.
Empirical Discoveries:
- Visual Quality $\neq$ Task Success: High aesthetic scores do not guarantee high task success rates; controllability (alignment between intended actions and predicted dynamics) is the critical factor.
- Data Scaling Law: Scaling post-training with action-observation data is more effective for embodied utility than simply upgrading the size of the pretrained video generator.
- Inference-Time Scaling: Allocating more compute to the planning phase (increasing the number of simulated rollouts per decision) substantially improves closed-loop performance.

Results and Analysis

The authors evaluated a wide range of models, including SVD, LTX-Video, Hunyuan, Wan2.1/2.2, Cosmos-P2, and proprietary models like Runway Gen4.

Performance Gains: Integrating world models consistently improved the performance of base proposal policies (e.g., VLMs or heuristics). For instance, in Active Recognition, the best post-trained model (Runway Gen4) achieved 64.79% accuracy with a mean trajectory of 4.06 steps, significantly outperforming the VLM baseline (50.27% accuracy, 6.24 steps).
Post-Training Efficacy: Post-trained variants (denoted with †) consistently outperformed their zero-shot counterparts. For example, post-trained Wan2.1 improved Active Recognition accuracy from 58.26% to 62.61% and ImageNav success rate from 38.19% to 45.14%.
Manipulation Challenges: Gains were less pronounced in robotic manipulation tasks. While post-training improved results (e.g., SVD† reached 46.5% success rate vs. 44.5% baseline), the difficulty of modeling contact-rich physics and precise kinematics remains a bottleneck compared to navigation tasks.
Scaling Trends:
- Data Scaling: Increasing post-training data from 400 to 80,000 instances consistently improved success rates, revealing a clear scaling law for adaptation.
- Inference Scaling: Increasing the number of world model inferences per episode (from 3 to 11) led to a monotonic increase in success rates, demonstrating the value of "thinking" (simulating futures) before acting.

Significance

The paper argues that the field of world models must shift its evaluation paradigm from generative quality to embodied utility. The authors claim that "world models live and die by their closed-loop success, not flawless generated visuals." By establishing World-in-World, the work provides a necessary framework to track genuine progress in embodied AI, demonstrating that while current models struggle with fine-grained physical dynamics, they can significantly aid perception and navigation when properly adapted and integrated into closed-loop planning systems. The findings suggest that future progress relies less on scaling pretraining data alone and more on action-conditioned post-training and inference-time computation to enhance controllability and decision-making.