🤖 AI

World Models for Embodied Intelligence: From Plausible to Controllable to Actionable

This paper proposes a new framework for evaluating world models in embodied intelligence that shifts focus from visual fidelity to a three-tiered hierarchy of Plausible, Controllable, and Actionable capabilities, arguing that true value lies in predictions that capture task-relevant structure, reflect intervention effects, and measurably improve closed-loop agent behavior.

Original authors: Nanjie Yao, Hao Wang, Chong Cheng, Zhikang Chen, Wenzhe Li, Jiafei Lyu, Li Shen, Peilin Zhao, Zongqing Lu, Gao Huang, Steven Hoi, Dacheng Tao, Deheng Ye

Published 2026-09-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nanjie Yao, Hao Wang, Chong Cheng, Zhikang Chen, Wenzhe Li, Jiafei Lyu, Li Shen, Peilin Zhao, Zongqing Lu, Gao Huang, Steven Hoi, Dacheng Tao, Deheng Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick up a cup. Before its fingers even touch the ceramic, a human mind has already anticipated the cup's weight, the friction of the surface, and the slight resistance of the handle. This mental simulation allows us to adjust our grip instantly, preventing a spill before it happens. For decades, scientists building artificial intelligence have tried to give machines this same ability to look ahead. They call these internal simulations "world models." The idea is simple: if a machine can build a mental picture of how the world changes when it acts, it can plan better, avoid mistakes, and learn faster. However, a new perspective suggests that simply making these mental pictures look realistic is not enough. A model might generate a beautiful, photorealistic video of a robot arm moving, yet fail to understand that the arm would actually knock the cup over. The true value of a world model lies not in how pretty its predictions are, but in whether those predictions help the machine make better decisions.

A comprehensive new survey by researchers from institutions including the Hong Kong University of Science and Technology, Tsinghua University, and Nanyang Technological University reorganizes how we think about these systems. Instead of sorting them by the complex computer code they use or the specific jobs they perform, the authors propose a new way to measure them based on what they can actually do. They introduce a three-step ladder of capability. At the bottom is the "Plausible" model. This level is satisfied if the machine can maintain a consistent story of the world over time. If a robot moves a block, a plausible model remembers that the block is still there and hasn't vanished or changed shape. It keeps the basic rules of geometry and physics intact, ensuring the mental picture doesn't drift into nonsense.

The next rung up is the "Controllable" model. This is where the machine learns to understand cause and effect. It is not enough for the model to just watch the world; it must understand how its own actions change that world. If the robot pushes a block to the left, a controllable model predicts that the block will move left, and that nothing else in the scene will suddenly shift. If the robot pushes it to the right, the prediction must change accordingly. The researchers emphasize that a model is only truly controllable if it can distinguish between different actions and show the specific, measurable difference each one makes. This is a critical step because it moves the machine from passively observing to actively understanding how to intervene.

The top of the ladder is the "Actionable" model. This is the ultimate goal: a system where the mental simulation directly improves the robot's performance in the real world. An actionable model does not just predict the future; it uses that prediction to choose a better path, learn a new skill faster, or recover from a mistake. For example, if a robot is navigating a room and its plan leads to a collision, an actionable model would spot the danger in its simulation before the robot actually crashes, allowing it to switch to a safe route. The survey finds that while many current systems are good at being plausible, and some are becoming controllable, very few have fully mastered the actionable stage where the simulation reliably leads to measurable success in complex, real-world tasks.

The researchers also mapped out how these models interact with the rest of the robot's brain. They identified four main ways a world model can help a machine improve. First, it can help gather better data by suggesting which experiments to run next. Second, it can act as a judge, scoring how well a plan will work before the robot tries it. Third, it can serve as a training ground, letting the robot practice millions of times in a safe, virtual environment. Finally, it can act as a safety net, constantly checking the robot's actions against its predictions and stepping in to correct course if reality starts to diverge from the plan.

This framework was applied to a wide range of robotic tasks, from delicate manipulation like picking up objects, to driving cars, to navigating through unknown buildings. The survey reveals that different tasks require different kinds of "grounding." A robot picking up a cup needs to understand physical forces like friction and weight. A self-driving car needs to understand the intentions of other vehicles and the geometry of the road. A walking robot needs to understand balance and terrain. The authors argue that a model that works well for one task might fail at another if it doesn't understand the specific rules of that environment.

Despite the progress, the paper highlights significant hurdles that remain. One major challenge is keeping the mental picture consistent over long periods. If a robot walks around a room for a long time, its internal map can start to drift, making objects appear in the wrong places. Another challenge is testing whether the model truly understands cause and effect, rather than just memorizing patterns from its training data. The researchers also point out that many current systems are too slow to be useful for fast-moving tasks, and that it is difficult to verify if a model is truly safe before deploying it in the real world.

The survey concludes that the field needs to shift its focus. Instead of celebrating how realistic a generated video looks, the community should prioritize whether the predictions lead to better, safer, and more efficient behavior. By organizing the field around these three levels of capability—plausibility, control, and actionability—the authors provide a clear roadmap for the next generation of embodied intelligence. The goal is no longer just to build a machine that can imagine the future, but to build one that can use that imagination to act wisely in the present.

Technical Summary: World Models for Embodied Intelligence: From Plausible to Controllable to Actionable

Problem Statement

Research in embodied intelligence is increasingly dominated by Vision-Language-Action (VLA) models that map multimodal observations directly to robot actions. While these models offer broad task coverage, they often lack explicit predictive mechanisms to maintain task-relevant state, anticipate delayed consequences, or compare alternative interventions. This limitation is critical in partially observed, contact-rich, and long-horizon environments where agents must reason about unobserved states and revise behavior when execution diverges from expectations.

Existing surveys on world models typically organize literature by architecture, output modality (e.g., video generation), or application domain (e.g., autonomous driving). These perspectives leave a central question implicit: Which predictive capabilities actually improve embodied behavior? The field lacks a unified framework that distinguishes between models that merely reconstruct the world plausibly and those that enable controllable intervention and measurable decision utility.

Methodology and Framework

The paper proposes a decision-centered framework to organize and evaluate world models, moving beyond architectural taxonomy to a capability-based hierarchy. This framework consists of two primary components:

1. The Capability Ladder

The authors define three progressively stronger capability levels, where higher levels imply the requirements of lower levels but add specific functional demands:

Plausible Models: These models preserve task-relevant temporal, geometric, or physical structure over time. The focus is on state consistency (identity, pose, occupancy, occlusion) and physical regularity (collision, contact, material response) rather than visual realism alone. Evaluation involves measuring error in task-relevant state variables across a rollout horizon.
Controllable Models: These models predict how interventions alter the preserved structure. A controllable model must demonstrate that different actions induce distinct, grounded changes in the predicted future (intervention fidelity). This requires testing counterfactuals where the initial state is held constant, but the action varies, ensuring the model responds correctly to commanded changes rather than just generating smooth, realistic sequences.
Actionable Models: These models translate predictions into measurable gains in downstream decision-making, learning, evaluation, verification, or recovery. A model is actionable only if its use improves a measured outcome (e.g., success rate, safety, sample efficiency) compared to a matched baseline without the model, under fixed computational and data budgets.

2. The Grounding–Improvement Matrix

To complement the ladder, the authors introduce a $3 \times 4$ matrix that links predictive content to system improvement mechanisms:

Grounding Dimensions (Rows):
- Geometry: Structure, layout, occupancy, 3D/4D consistency.
- Physics: Dynamics, contact, force, material response, executability.
- Action: Interventions, semantics, control, causality.
Improvement Loops (Columns):
- Data Loop: Acquiring or generating informative transitions.
- Reward Loop: Evaluating outcomes to shape rewards or critics.
- Policy Loop: Using predictions for planning, search, or policy learning.
- Model-Self Loop: Updating the predictor based on verification errors.

This matrix allows for a fine-grained analysis of how specific grounded predictions (e.g., contact forces) interact with specific system components (e.g., policy updates) across different domains.

Key Contributions

Unified Capability Ladder: The paper formulates "Plausible," "Controllable," and "Actionable" as a decision-centered hierarchy. This provides a common language to evaluate methods across diverse architectures (latent dynamics, video generation, object-centric models) and domains (manipulation, driving, locomotion).
Grounding–Improvement Matrix: It connects the content of predictions (geometry, physics, action) with the mechanism of improvement (data, reward, policy, model-self), offering a structured way to analyze how world models influence embodied agents.
Comprehensive Technical Survey: The authors organize representative methods into coherent progressions within each capability level, analyzing their cross-cutting connections. This covers manipulation, navigation, locomotion, autonomous driving, and general embodied learning, tracing technical advancements from passive prediction to closed-loop decision support.
Evaluation Protocols and Open Problems: The paper outlines evaluation protocols aligned with the three capability levels, emphasizing the need for matched baselines, intervention tests, and closed-loop utility metrics. It identifies critical open problems, including long-horizon consistency, uncertainty calibration, causal intervention testing, latency constraints, and cross-embodiment transfer.

Results and Findings

The survey synthesizes the current state of the field through the proposed framework:

Plausibility: Progress has moved from pixel-based prediction to compact latent and object-centric states, and further to metric scene states (occupancy, point clouds) and physics-grounded dynamics. However, long-horizon stability and persistent memory remain challenges, with many models suffering from drift or hallucination over extended rollouts.
Controllability: The field has expanded from simple action-conditioned prediction to open, latent, and language-specified interventions. Recent work focuses on "structure-grounded control," where interventions are specified in measurable geometric or physical variables (e.g., camera trajectories, contact forces) to enable rigorous testing of intervention fidelity. Joint world-action models are emerging to couple prediction with action generation.
Actionability: Actionable models are demonstrated in four functional stages: (1) planning and value-based selection (scoring candidate behaviors), (2) action decoding from imagination (generating actions directly from predicted futures), (3) learned simulators for policy training and evaluation, and (4) runtime verification and recovery (detecting and correcting execution errors). The survey notes that while many models claim utility, few provide rigorous evidence of measurable downstream gains under matched budgets.
Domain Specificity: Different embodiments prioritize different variables. Manipulation relies on contact and object state; autonomous driving on multi-agent futures and safety; navigation on persistent geometry and belief; and locomotion on fast dynamics and stability. The framework highlights that a method successful in one domain may not transfer directly to another without addressing these specific variable requirements.

Significance and Claims

The paper claims that the value of a world model for embodied intelligence lies not in the fidelity of its visual reconstruction, but in its ability to improve agent behavior. By shifting the evaluation focus from "visual plausibility" to "task-relevant state preservation," "intervention fidelity," and "closed-loop decision utility," the authors argue that the field can better identify which predictive capabilities are truly necessary for robust embodied intelligence.

The framework serves as a diagnostic tool to:

Clarify the gap between current methods and the requirements for fully controllable and actionable agents.
Standardize evaluation by requiring matched baselines and specific intervention tests rather than aggregate visual metrics.
Guide future research toward solving bottlenecks such as uncertainty calibration, causal action effects, and the transfer of capabilities across different embodiments.

The authors conclude that progress toward "Actionable" world models must be evaluated by whether grounded predictions improve closed-loop behavior under explicit data, compute, and latency budgets, rather than by the quality of generated video alone.