💻 computer science

iMaC: Translating Actions into Motion and Contact Images for Embodied World Models

This paper introduces iMac, a novel embodied world model paradigm that replaces low-dimensional structured action vectors with raw visual images as native action tokens to achieve superior prediction accuracy, task success, and generalization across diverse robotic embodiments.

Original authors: Zhenyu Wu, Xiuwei Xu, Yukun Zhou, Yifan Li, Qiuping Deng, Xiaofeng Wang, Zheng Zhu, Bingyao Yu, Ziwei Wang, Jiwen Lu, Haibin Yan

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenyu Wu, Xiuwei Xu, Yukun Zhou, Yifan Li, Qiuping Deng, Xiaofeng Wang, Zheng Zhu, Bingyao Yu, Ziwei Wang, Jiwen Lu, Haibin Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fold a shirt or pick up a cup. To do this safely and efficiently, you want to test the robot's brain (its "policy") thousands of times before letting it touch a real object. Usually, you'd have to build a physical simulation, which is hard, or just let the robot fail in the real world, which is slow and risky.

This paper introduces iMaC (Images of Motion and Contact), a new kind of "crystal ball" for robots. Instead of just guessing what the future looks like, iMaC acts like a highly accurate movie director that simulates exactly what will happen if a robot moves its arm in a specific way.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Black Box" of Robot Actions

Most current robot simulators are like a magician who waves a wand and says, "I will move the robot." They take a robot's command (like "move arm 5cm forward") and turn it into a tiny, abstract code number. They then feed this number into a video generator and hope the robot's arm actually moves correctly in the video.

The problem is that in the real world, being off by just a few centimeters can mean the difference between grasping a cup and knocking it over. Abstract codes are too vague to capture these tiny, critical details. The simulator often has to "guess" (or hallucinate) where the robot's hand ends up, leading to errors that pile up over time.

2. The Solution: Drawing the Future Instead of Guessing It

iMaC changes the game by refusing to guess. Instead of using abstract codes, it draws the future for the robot. It translates the robot's commands into actual pictures that show exactly where the robot's body will be and how close it will get to objects.

It does this using two special "instruction manuals" (which the paper calls Motion Images and Contact Images):

Motion Images (The "Where"):
Think of this as a blueprint. Before the video starts, iMaC uses the robot's official instruction manual (called a URDF) to calculate exactly where every joint will be. It then renders a video of the robot's arm moving through space, as if it were a 3D animation.
- Analogy: Instead of telling a painter, "Draw a car moving," iMaC hands the painter a photo of the car already in the exact spot it needs to be. The video generator just has to fill in the background.
Contact Images (The "Touch"):
This is the secret sauce. It's like a heat map showing the distance between the robot's hand and the objects in the room.
- Analogy: Imagine a "force field" visualization. One map shows how close the robot's hand is to a table (Robot-to-Scene), and another shows how close the table is to the robot's hand (Scene-to-Robot). This tells the simulator, "Hey, the hand is this close to the cup; if it moves 1mm more, it will touch." This ensures the simulation knows exactly when a collision or grasp happens.

3. The "Self-Teaching" Loop

When you watch a long movie, the end of one scene becomes the start of the next. In robot simulations, if the model makes a tiny mistake in the first second, that mistake gets bigger and bigger by the end of the minute.

iMaC solves this by practicing what it preaches. During its training, it doesn't just look at perfect, real-world videos. It generates a chunk of video, takes the end of that generated video, and uses it as the start for the next chunk.

Analogy: It's like a student who practices by taking a test, grading their own answers, and then using those (sometimes imperfect) answers as the starting point for the next practice test. This teaches the model to handle its own mistakes so it doesn't get confused when running a long simulation in the real world.

4. The Results: A Better "Test Drive"

The researchers tested iMaC on eight difficult real-world robot tasks (like moving objects or manipulating tools). They used it to "test drive" different robot brains (policies) to see which ones were better.

The Finding: The scores iMaC gave to the robot brains matched the robots' actual performance in the real world almost perfectly.
Why it matters: If a robot brain performs well in the iMaC simulation, it will almost certainly perform well in the real world. This means engineers can safely and quickly pick the best robot "brain" without needing to run thousands of expensive, physical trials.

Summary

In short, iMaC is a robot simulator that stops guessing and starts showing. By turning robot commands into clear pictures of movement and distance, it creates a highly reliable "virtual reality" where we can test robot skills safely, knowing that if it works in the simulation, it will work in reality.

Technical Summary: iMaC – Translating Actions into Motion and Contact Images for Embodied World Models

1. Problem Statement

Embodied world models aim to serve as real-world simulators for robot policy evaluation and closed-loop rollout, offering a scalable alternative to expensive and unsafe physical trials. However, the reliability of these models hinges on their ability to condition future video predictions precisely on robot actions.

Existing action-conditioned video models typically encode future actions as compact vectors, injecting them via learned modules (e.g., cross-attention, AdaLN, FiLM). This abstraction forces the model to indirectly infer fine-grained spatial consequences. In real-world manipulation, where centimeter-level differences determine contact, object motion, and task success, such indirect inference is insufficient. Furthermore, existing explicit representations (e.g., projected spheres or action maps) often visualize actions but fail to directly control the future robot body state or explicitly describe the interaction geometry between the future robot and the current scene.

2. Methodology

The paper proposes iMaC (images of Motion and Contact), an embodied world model that translates future actions into dense, image-like controls to guide video generation. The approach is built upon a WAN2.2 Image-to-Video (IT2V) DiT backbone and operates through the following key components:

2.1 Motion Images (Robot Kinematics)

To eliminate the need for the model to hallucinate robot motion from abstract vectors, iMaC converts future joint action sequences into Motion Images.

Mechanism: Using the robot's URDF and forward kinematics, the system renders the future robot configurations from the specific camera views (head and wrist cameras) used in the observation.
Function: These rendered videos serve as dense control signals that specify the future robot's visual state (appearance and pose) directly in pixel space, reducing the burden on the generative model to infer robot geometry.

2.2 Contact Images (Two-Stream Geometry)

To capture how robot motion interacts with the scene, iMaC constructs Contact Images based on 3D pointclouds. This involves an auxiliary depth prediction branch where depth maps are encoded as colorized images to align with the RGB input.

Scene-to-Gripper Stream ( $C_{s \to g}$ ): Calculates the distance from current scene points to the future gripper position. This encodes which parts of the current environment are close to the future end-effector.
Robot-to-Scene Stream ( $C_{r \to s}$ ): Calculates the distance from future robot points to the current scene. This encodes the spatial relation of the future robot body relative to the static environment.
Integration: These distance maps are projected and densified into image-like controls, providing explicit guidance on contact-relevant spatial relations.

2.3 Training-Time Rollout Strategy

To support long-horizon manipulation and mitigate exposure bias (where the model is trained on ground-truth references but tested on its own imperfect generations), iMaC employs a training-time rollout strategy.

During training, the model generates a chunk of video, decodes the final frame, and uses this generated frame as the reference input for the subsequent chunk.
This allows the model to learn to construct subsequent geometric states from its own predictions rather than relying solely on ground-truth references, better aligning training conditions with closed-loop inference.

3. Key Contributions

Spatially Explicit Action Conditioning: Unlike methods using compact action vectors, iMaC translates actions into dense, image-like controls (Motion and Contact images) that explicitly define future robot appearance and robot-scene geometry.
Two-Stream Geometry Controls: The introduction of bidirectional distance maps (Scene-to-Gripper and Robot-to-Scene) provides the world model with precise cues for contact and interaction dynamics, which are critical for manipulation tasks.
Robust Long-Horizon Evaluation: The training-time rollout strategy effectively reduces the train-test mismatch in closed-loop generation, enabling minute-level video generation for policy evaluation.
Policy Evaluation Capability: The framework demonstrates the ability to rank different policy checkpoints based on world-model success estimates, showing strong correlation with real-world performance.

4. Experimental Results

The authors evaluated iMaC on eight challenging, long-horizon real-robot manipulation tasks involving multi-view RGB observations.

Video Prediction Quality: iMaC achieved state-of-the-art performance in task-averaged metrics (FID, PSNR, SSIM, FVD) compared to baselines like Ctrl-World and ABot-PhysWorld, demonstrating that motion and contact images significantly improve action-conditioned prediction.
Policy Evaluation Correlation: In closed-loop rollouts, iMaC's success estimates showed a strong positive correlation ( $r \in [0.833, 0.956]$ ) with real-world success rates across six of the eight tasks. This indicates the model can effectively preserve the relative ranking of different policy families and training checkpoints.
Ablation Studies:
- Removing Contact Images resulted in the model failing to grasp objects correctly, despite generating plausible interaction-like motions, highlighting the necessity of contact-aware guidance.
- Removing Motion Images caused the model to fail in producing precise grasps, as it lacked direct guidance on future robot configurations.
- Using external depth (DA3) instead of the model's own predicted depth for contact construction led to less consistent contact geometry and missed grasps.

5. Significance and Claims

The paper claims that iMaC addresses a critical bottleneck in embodied world models: the lack of spatial precision in action conditioning. By converting actions into explicit visual and geometric controls, iMaC enables reliable simulation of manipulation tasks where fine-grained spatial relationships dictate success.

The primary significance lies in its application as a policy evaluator. The authors demonstrate that iMaC can serve as a scalable, in-silico simulator to compare policy checkpoints and estimate real-world performance without requiring task-specific physical simulators or extensive hardware trials. The strong correlation between world-model scores and real-world outcomes suggests that such models can effectively guide checkpoint selection and policy improvement, provided the tasks do not suffer from severe missing-observation issues (e.g., tasks heavily dependent on height relations not visible to the cameras).