Technical Summary: Physically Viable World Models: A Case for Query-Conditioned Embodied AI

1. Problem Statement

Current world models for embodied AI are primarily optimized to predict future observations from past data. While these models often produce visually plausible rollouts, they frequently fail to represent the underlying physical structure required to answer intervention queries. The core issue is structural: distinct physical systems can generate identical observations yet behave differently under intervention (e.g., varying mass, friction, viscosity, or restitution).

Consequently, observation-predictive models may recommend infeasible actions, mispredict interaction outcomes, or certify behaviors as safe that would be unsafe in reality. This failure is not merely a matter of insufficient data or architectural scale; it stems from the inability of current training regimes to uniquely identify the latent physical variables (such as density or contact state) that determine the outcome of an action when those variables are not directly observable. The paper argues that embodied AI requires world models that are physically viable: constructed to answer specific intervention queries by representing the minimal physical abstraction sufficient to preserve the distinctions relevant to that query.

2. Methodology

2.1 Controlled Evaluation Suite

To expose the limitations of current observation-predictive models, the authors constructed a suite of controlled benchmarks where the visible scene remains fixed while latent physical parameters vary. The evaluation targets three model families:

1. Vision-Language Models (VLMs): Tested on static image prediction of physical outcomes.
2. Video Diffusion Models: Tested on generating video continuations from keyframes.
3. Action-Conditioned Latent Predictors: Tested on planning and control tasks (using V-JEPA 2).

The benchmark scenarios include:

• Rigid-Body Interactions: A ball rolling down a ramp to hit block towers of varying density (wood vs. steel) or restitution (bouncy vs. rigid).
• Deformable Interactions: Collisions with jelly walls of varying stiffness and configuration.
• Rigid-Fluid Coupling: A ball striking a cup filled with varying levels of liquid (water vs. honey-like viscosity).
• Contact-Dependent Pushing: A robot arm pushing a wall at different heights or with different floor friction coefficients.
• Viscosity-Dependent Pouring: A robot pouring liquids of varying viscosities, requiring different hold durations and motion profiles to achieve a target volume.

2.2 Proposed Framework: Query-Conditioned Construction

The paper proposes a shift from "observation extrapolation" to query-conditioned construction of physically viable world models. This approach relies on an autonomous orchestrator that dynamically assembles a world model for a specific query. The construction process involves:

1. Query Analysis: The orchestrator identifies the physical distinctions required to answer the query (e.g., does the task depend on mass, friction, or fluid inertia?).
2. Abstraction Selection: It selects the simplest set of variables, parameters, and constraints sufficient to preserve those distinctions. This avoids the need for a universal, high-fidelity simulator.
3. Component Composition: The model is composed of modular, compatible components:
   • Perception & Representation: Maps sensor data (images, LiDAR, force-torque) to physical variables (pose, velocity, material properties).
   • State & Parameter Estimation: Estimates latent quantities (e.g., viscosity, mass) or maintains uncertainty bounds if they cannot be identified.
   • Action Specification: Defines admissible interventions in the selected variable space.
   • Dynamics & Constraints: Evolves the state under intervention using analytic equations, numerical solvers (e.g., MPM for fluids), learned surrogates, or hybrids, ensuring the physical structure determining the outcome is preserved.
   • Query-Level Response: Returns the specific answer required (e.g., a trajectory, a reachability set, a parameter estimate, or a safety certificate).

The orchestrator acts as a closed-loop selection process, routing between analytic, numerical, and learned models, and revising the construction if components fail compatibility checks or if the query changes.

3. Key Contributions

1. Structural Argument: The paper argues that observation-predictive world models are structurally inadequate for embodied intervention because they conflate perceptual regularities with physical dynamics. It posits that embodied world models must be constructed per query, centered on the physical distinctions the intervention depends on.
2. Empirical Demonstration of Failure: Through controlled experiments, the authors demonstrate that VLMs, video diffusion models, and latent predictive models fail to answer queries correctly when latent physical variables (density, restitution, viscosity, contact height) change while visual appearance remains constant.
   • VLMs often miss thresholded outcomes or construct qualitatively incorrect scenarios under low context.
   • Diffusion models produce visually coherent videos that violate contact, fluid, or deformable dynamics.
   • Latent predictors generate visually plausible trajectories that are physically infeasible (e.g., ignoring friction or torque limits).
3. Modular Design Framework: The paper defines a modular framework for constructing physically viable world models. It specifies the operations an orchestrator must perform to compose these models, emphasizing that the "right" abstraction is not the most detailed model, but the simplest one that preserves the distinctions relevant to the query.

4. Results

The experimental results highlight systematic failure modes across different model families:

• Visual Plausibility vs. Physical Validity: Models often generate rollouts that look realistic but fail to respect physical laws (e.g., a ball passing through a cup, or a jelly wall behaving like a rigid block).
• Latent Variable Sensitivity: When physical parameters like material density or fluid viscosity are altered, observation-predictive models fail to adjust their predictions accordingly, often defaulting to the most visually similar training distribution.
• Control Failures: In planning tasks, action-conditioned models optimize for latent visual similarity rather than physical feasibility, leading to trajectories that fail when executed in a physics simulator (e.g., a robot arm pushing a wall at a height that causes tipping rather than sliding, or failing to account for wall material properties).

In contrast, the proposed query-conditioned approach successfully addresses these failures by explicitly representing and estimating the relevant latent variables (e.g., inferring viscosity from a probing pour before planning the main pour) and using appropriate dynamics (e.g., MPM simulation for fluid interaction).

5. Significance and Claims

The paper claims that physical viability is a necessary property for world models used in planning, control, counterfactual reasoning, and safety analysis. Its significance lies in:

• Redefining the Goal: Shifting the objective from "predicting the next observation" to "answering intervention queries with physical consistency."
• Design Principle: Establishing that the correct world model is the simplest abstraction sufficient for the query, rather than a maximally detailed simulation.
• Interpretability and Verifiability: The modular decomposition makes models interpretable, their components verifiable, and their outputs auditable against the query.
• Feasibility Test: Providing a criterion to evaluate existing models: do they preserve the physical distinctions relevant to the intervention?

The authors acknowledge that creating an autonomous orchestrator capable of identifying the right abstraction and composing components remains an open research problem. They also note that physical viability does not overcome fundamental identifiability limits; latent quantities may still require active interaction or conditional responses if they cannot be resolved from passive observation. However, the paper asserts that this approach identifies a class of failures that cannot be resolved through scaling alone and motivates architectures that explicitly represent actions, state, dynamics, and constraints.