🤖 AI

WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation

This paper introduces WorldSimProbe, a capability-based diagnostic framework that formalizes an "Observable Simulator Contract" to rigorously evaluate the fidelity of action-conditioned world models through controlled suites assessing motion induction, interaction grounding, and dynamics, revealing systematic failures in existing models that traditional visual or task-based metrics overlook.

Original authors: Peterson Co, Sicheng Hu, Chunxuan Jiao, Hongyang Cheng, Yulin Luo, Yijie Xu, Sixiang Chen, Zhongxia Zhao, Zihao Wang, DaFeng Chi, Peidong Liu, YuTong Chen, Henghua Liu, Zhihao Yuan, Huizhu Jia, Yuzhen

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peterson Co, Sicheng Hu, Chunxuan Jiao, Hongyang Cheng, Yulin Luo, Yijie Xu, Sixiang Chen, Zhongxia Zhao, Zihao Wang, DaFeng Chi, Peidong Liu, YuTong Chen, Henghua Liu, Zhihao Yuan, Huizhu Jia, Yuzheng Zhuang, Tianle Zhang, Liang Lin, Huajie Tan, Shanghang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a sandwich. You don't just want the robot to look like it's making a sandwich; you need it to actually pick up the bread, spread the peanut butter, and not accidentally throw the jar across the room. In the world of artificial intelligence, scientists are building "world models"—think of them as a robot's internal daydream machine. These models try to predict what will happen next if the robot moves its arm in a certain way. For a long time, researchers mostly checked if these daydreams looked pretty and realistic, like a high-quality movie. But there's a catch: a movie can look amazing while breaking the laws of physics. If a robot's brain thinks it can push a cup but the cup doesn't move, or if it thinks it grabbed a spoon when it actually just waved near it, the robot will fail in the real world. This paper tackles the tricky question of how to tell if a robot's "daydream" is actually a faithful simulation of reality, or just a convincing lie.

The authors of this paper, a team from various universities and research labs, realized that current tests for these AI models were too easy. They were mostly asking, "Did the robot finish the task?" or "Does the video look good?" But they wanted to ask harder questions: "If I wiggle the robot's arm just a tiny bit, does the simulation wiggle the same way?" or "If I tell the robot to tap a glass, does it actually make contact, or does it just pretend to?" To answer this, they created a new diagnostic tool called WorldSimProbe. Think of it as a rigorous "lie detector test" for robot simulators. Instead of just watching the final movie, they poke and prod the AI's predictions at every step of the way to see if the cause-and-effect chain holds up.

They put six different open-source AI models through their paces using over 18,000 test cases across three different robotic environments. The results were eye-opening. They found that while these AI models are great at generating smooth, plausible-looking videos, they often fail the "physics test." For instance, when the researchers gave the models slightly different instructions, many models didn't change their predictions enough, acting like they were stuck in a loop rather than reacting to the new input. Even worse, the models frequently hallucinated interactions. They would show a robot "grabbing" an object that was actually too far away, or making a heavy object float when it should have stayed put. The paper explicitly argues against the idea that a successful-looking video means the model is faithful; a model can get the right answer (the task is done) for the wrong reasons (the physics were fake).

The study didn't just find problems; it categorized them. They discovered that models struggle most with "interaction grounding"—knowing when an object is actually being touched versus just being near it—and "dynamics," which is how objects move and react after being hit or pushed. For example, the models were surprisingly bad at simulating a "shake" or a "tap," often getting the motion completely wrong. However, they did find that the models were better at preserving the style of movement if they were trained on human data, suggesting that copying human motion helps, but doesn't fix the underlying physics issues.

Ultimately, the paper suggests that we need to stop just grading these AI models on how pretty their videos are. The authors propose a new standard called the "Observable Simulator Contract," which demands that if you give a robot an action, the simulation must show the exact physical motion that action causes, and the environment must react only to that motion. Their tests show that current models are far from perfect at this, often failing in systematic ways that could lead to real-world robots crashing or dropping things. By using WorldSimProbe, researchers can now pinpoint exactly where a model is lying to us, whether it's failing to react to small changes, pretending to touch things it can't reach, or getting the physics of a drop all wrong. This isn't just about ranking which AI is "best"; it's about building a transparent way to fix the cracks in the foundation before we let these robots loose in our homes and factories.

Technical Summary: WorldSimProbe

Problem Statement

Action-conditioned world models (ACWMs) are increasingly utilized to provide embodied AI with predictive simulators for planning, policy evaluation, and data generation. However, realizing their potential requires precise action-conditioned transitions rather than merely plausible visual outputs. Current evaluation paradigms suffer from significant limitations: they predominantly emphasize visual quality, task outcomes, or coarse rollout-level responsiveness. Crucially, existing benchmarks often fail to directly test simulator fidelity by examining whether predicted environment responses are causally supported by the motion induced by supplied actions. Consequently, a rollout may appear action-aware or physically plausible while lacking faithful action execution or physically grounded interactions. There is a need for a diagnostic framework that moves beyond aggregate success rates to systematically probe the specific capabilities required of physical simulators.

Methodology: The Observable Simulator Contract and WorldSimProbe

To address these gaps, the authors formalize the Observable Simulator Contract, a minimal requirement for any action-conditioned physical simulator. This contract defines two linked consistency conditions:

Action-Realization Consistency: The generated agent motion ( $\hat{r}_{t:t+H}$ ) must correspond to the supplied action stream ( $a_{t:t+H}$ ) under the current scene state.
Interaction-Response Consistency: The generated environment response ( $\hat{e}_{t:t+H}$ ) must be physically supported by the realized agent motion and the initial environment state.

Based on this contract, the authors introduce WorldSimProbe, a diagnostic benchmark comprising five controlled evaluation suites that progress from supplied action to realized motion and finally to environment response. These suites utilize simulator-executable interventions and targeted evaluators to localize failures:

Local Action Calibration (Task 1): Tests sensitivity to fine-grained action perturbations. By applying small and large perturbations to action dimensions while holding task semantics fixed, the suite measures whether the model's response scaling matches the simulator's calibrated response. It uses a ratio of Mean Squared Error (MSE) to evaluate relative response scaling, reducing sensitivity to model-specific visual error baselines.
Global Trajectory Coverage (Task 2): Evaluates action realization beyond task-associated trajectories using cross-task "receiver–donor" replay. Actions sampled from a donor task are applied to a receiver scene. The suite employs Robot-Masked Flow Alignment (RMFA) to measure dense optical flow alignment between generated and simulator-reference rollouts, specifically focusing on active robot-arm pixels.
Action-Source Behavior Preservation (Task 3): Tests whether action realization preserves source-specific behaviors (e.g., variations in speed, smoothness, or execution strategy from human teleoperators or different policy checkpoints). It assesses if models revert to canonical execution patterns rather than preserving the unique characteristics of the supplied action source.
Interaction Grounding (Task 4): Examines whether realized motion correctly triggers agent-environment interactions. The suite constructs controlled "no-contact" scenarios (e.g., spatial proximity without contact, visual distractors) to detect contact hallucinations, where models generate interactions unsupported by the supplied action and scene geometry. Success is measured by the lack of displacement in objects that should remain stationary.
Interaction Dynamics (Task 5): Evaluates the fidelity of dynamics when interactions do occur. Using eight primitive interaction types (push, pull, drag, rotate, shake, tap, knock-over, drop), a Vision-Language Model (VLM) judge predicts the realized primitive from the generated video and compares it against the ground-truth primitive defined by the simulator.

Experimental Setup

The authors evaluated six open-source ACWMs (IRASim, Ctrl-World, BWM, DreamDojo, LingBot-VA, and Cosmos-3-Nano) across three distinct simulators: RoboTwin, ManiSkill, and LIBERO. The benchmark comprises over 18,000 evaluation instances. Models were trained on official splits with modified action dimensions and input sizes to match simulator interfaces, while other settings remained at their released defaults.

Key Results

Overall Faithfulness (RQ1): Performance varies significantly across platforms and suites. No single model dominates all categories; for instance, LingBot-VA leads on RoboTwin and ManiSkill, while Ctrl-World leads on LIBERO. Crucially, strength in one stage of the simulator chain (e.g., action realization) does not guarantee fidelity in others (e.g., interaction dynamics), highlighting the necessity of multi-stage evaluation.
Action-to-Motion Robustness (RQ2):
- Local Calibration: Models exhibit varying degrees of attenuation in response to perturbations compared to the simulator oracle. Some models fail to detect action changes or fail to scale responses correctly, leading to early failure thresholds.
- Global Coverage: Action-realization fidelity declines as the mismatch between donor and receiver motion increases. Models appear to rely increasingly on scene- or task-associated motion priors when controls deviate from familiar trajectories.
- Source Preservation: Models struggle to preserve source-specific behaviors, particularly for human trajectories and early-policy checkpoints, often reverting to canonical execution styles.
Interaction Failures (RQ3):
- Grounding: Models are prone to "contact hallucination," particularly in scenarios with visual contact cues lacking control support (false contact), where grounding scores drop significantly compared to distractor or proximity cases.
- Dynamics: Fidelity varies more across interaction primitives than across models. "Tap" interactions show the highest fidelity, while "shake" is nearly absent (0.0–1.2% accuracy). Models share systematic biases in modeling specific dynamics.
Benchmark Validity (RQ4):
- Human Agreement: The RMFA metric used in WorldSimProbe shows strong alignment with human judgments ( $\rho = 0.750$ ), outperforming both binary VLM judgments and Inverse Dynamics Model (IDM) recovery errors, which suffer from inverse-model decoding failures.
- Downstream Utility: Policies trained on synthetic data generated by higher-fidelity models (as ranked by WorldSimProbe) demonstrate superior performance on Out-of-Distribution (OOD) controls, whereas success-oriented training on standard controls obscures these fidelity differences.

Significance and Contributions

The paper claims that WorldSimProbe provides a transparent and standardized paradigm for diagnosing ACWM simulator fidelity beyond coarse, task-directed evaluation. Its primary contributions are:

Formalization of the Observable Simulator Contract: Defining simulator faithfulness through observable correspondences between supplied actions, realized motions, interactions, and environment responses.
Diagnostic Benchmark: Introducing WorldSimProbe with five controlled suites and over 18,000 instances, enabling the localization of distinct failure modes (e.g., action calibration, unsupported interactions, inconsistent dynamics) rather than relying on aggregate rollout scores.
Systematic Diagnosis: Revealing systematic failures in existing ACWMs, including action compression, degraded generalization beyond task-specific controls, and structured failures in interaction grounding and dynamics.

The authors conclude that while current models can generate plausible futures, they often lack the faithful simulation of action-conditioned transitions required for reliable embodied manipulation. WorldSimProbe identifies where, when, and how simulator fidelity breaks down, providing a foundation for targeted improvements in ACWMs as embodied simulators. Future work is noted to potentially extend this framework to real-world rollouts, longer-horizon predictions, and deformable object interactions.