🤖 AI

Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

Mask2Real-WM is a two-stage action-conditioned world model for dexterous manipulation that bridges the sim-to-real gap by decoupling dynamics prediction in segmentation space from photorealistic rendering, enabling precise per-joint control through large-scale synthetic pretraining and minimal real-world fine-tuning.

Original authors: Riccardo O. Feingold, Davide Liconti, Chenyu Yang, Robert K. Katzschmann

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Riccardo O. Feingold, Davide Liconti, Chenyu Yang, Robert K. Katzschmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the factory floor, moving with precision in environments where every object is fixed and every path is pre-programmed. But the moment a robot steps into a messy, unpredictable home or a cluttered workshop, its confidence often evaporates. To navigate the real world, a machine needs more than just sensors; it needs a sense of what will happen next. This is the promise of "world models," a concept in robotics where an artificial intelligence learns to imagine the future. Instead of just reacting to what it sees right now, the robot builds a mental simulation of how the world changes when it moves its arm or grasps an object. If a robot can reliably predict the outcome of its actions in its own mind, it can test different strategies without risking damage to itself or its surroundings, effectively "dreaming" up the best way to solve a problem before it ever touches a physical object.

The challenge becomes exponentially harder when the robot has a hand with many moving parts, like a human hand, rather than a simple two-fingered gripper. These dexterous hands offer incredible versatility, capable of manipulating delicate objects or performing complex tasks, but they also introduce a dizzying number of variables. A single hand might have twenty-three different joints that can move independently, creating a vast space of possible actions. Predicting how such a complex system will interact with the world is difficult because the visual changes are subtle and intricate. A slight shift in a finger can drastically alter how an object is held, yet standard video prediction models often blur these fine details, failing to capture the precise cause-and-effect relationship between a specific joint movement and the resulting motion of an object.

Researchers at ETH Zurich have developed a new approach to this problem, creating a system they call Mask2Real-WM. Their goal was to build a world model that could accurately predict the future of a dexterous robot hand interacting with objects, even when trained on a very small amount of real-world data. The core of their innovation lies in how they break down the prediction task. Instead of trying to predict the full, photorealistic video of the future all at once, they split the process into two distinct stages. The first stage focuses purely on the structure of the scene, predicting a simplified map of where the hand and the object will be. The second stage takes that map and paints it into a realistic video.

This separation is crucial because it allows the researchers to use a massive amount of simulated data to teach the robot how things move, while using only a tiny amount of real-world footage to teach it how things look. In the first stage, the system learns to predict "segmentation masks," which are essentially color-coded outlines showing the hand, the object, and the background. Because these outlines are simple shapes rather than complex photographs, the gap between what a computer simulation looks like and what the real world looks like is much smaller. This allows the researchers to train the movement-prediction part of the system on over fifty hours of synthetic data generated in a computer simulation. In this virtual environment, the robot can practice moving its fingers in every possible way, covering a range of motions that would take years to collect in a real lab.

Once the system has learned the physics of movement from this vast library of simulations, it is fine-tuned with just two and a half hours of real-world video. This is where the second stage comes in. The system takes the predicted outlines from the first stage and uses a specialized rendering model to fill them in with realistic colors, textures, and lighting. Because the heavy lifting of understanding motion was already done in the simulation, the rendering model only needs to learn the specific look of the real environment, a task it can master with very little data. The result is a system that can watch a robot hand move, imagine the next few seconds of action, and generate a photorealistic video of what will happen, all while keeping the precise movements of each finger sharp and distinct.

The researchers tested this system on a benchmark task involving picking up and placing small everyday objects. They found that the two-stage approach was far superior to previous methods that tried to predict the video in one single step. The older methods could often guess the general path of the hand, but they struggled with the fine details, blurring the fingers together or failing to show how a specific joint movement changed the grip. In contrast, the new system demonstrated a high degree of control: when the researchers asked the model to move just one specific finger, the model accurately predicted the movement of that single finger without confusing it with the others. This level of precision is essential for a robot to truly understand the consequences of its actions.

The study also revealed that the large-scale simulation training was not just a helpful bonus but a necessity. When the researchers tried to train the movement-prediction part of the system using only the small amount of real-world data, the system failed to learn the independent control of the fingers. It simply did not have enough examples of isolated finger movements to understand how they worked. The simulation data provided the missing variety, exposing the system to every possible angle and motion of the hand. By combining this broad simulated experience with a small dose of real-world visual training, the researchers created a model that could generalize to new situations, such as different lighting or objects it had never seen before, with a reliability that previous models could not match.

Ultimately, this work demonstrates a practical path forward for teaching robots to think before they act. By using simplified representations to bridge the gap between simulation and reality, the researchers have shown that a robot can learn complex physical interactions without needing thousands of hours of real-world trial and error. The system does not just guess what the next frame of a video will look like; it understands the underlying mechanics of the movement well enough to predict the future with confidence. This capability opens the door for robots to safely explore new tasks in their own minds, evaluating different strategies and learning from their imagined mistakes before ever engaging with the physical world.

Technical Summary: Mask2Real-WM

Problem Statement

Action-conditioned world models are essential for robot learning, enabling policy evaluation, planning, and data augmentation by predicting future states without physical interaction. However, building such models for dexterous manipulation remains a significant challenge. The primary hurdles include:

High-dimensional action spaces: Dexterous hands involve numerous degrees of freedom (e.g., 23 dimensions in this setup: 6-D Cartesian end-effector pose + 17 hand-joint positions).
Data scarcity: Large-scale, high-quality dexterous manipulation datasets are rare.
Prediction fidelity: Existing video models often struggle to accurately predict fine-grained finger-object contacts, frequently blurring these critical regions.
Sim-to-Real gap: Training on synthetic data often fails to transfer to reality due to discrepancies in appearance (lighting, texture) and dynamics.

Methodology: Mask2Real-WM

The authors propose Mask2Real-WM, a two-stage, action-conditioned world model that decouples pixel prediction into a dynamics model and a rendering model. This factorization allows the system to leverage the smaller sim-to-real gap inherent in segmentation masks while still producing photorealistic outputs.

1. Problem Formulation

The model operates in a multi-view setup ( $V=2$ ) with a context window of $k=5$ past observations and predicts a future horizon of $H=5$ steps. The goal is to model the joint distribution of future RGB frames ( $I$ ) and segmentation masks ( $m$ ) given past observations and an action sequence ( $a$ ):
$p(I_{t+1:t+H}, m_{t+1:t+H} \mid I_{t-k:t}, m_{t-k:t}, a_{t-k:t+H})$

2. Factorization Strategy

The joint distribution is decomposed into two independent stages:
$p(I, m \mid \dots) = \underbrace{p(m_{t+1:t+H} \mid m_{t-k:t}, a_{t-k:t+H})}_{\text{WM1: Dynamics}} \cdot \underbrace{p(I_{t+1:t+H} \mid I_{t-k:t}, m_{t-k:t+H}, a_{t-k:t+H})}_{\text{WM2: Rendering}}$

WM1 (Dynamics Model): Predicts future segmentation masks from past masks and the action sequence. Because segmentation masks have a smaller sim-to-real gap than RGB images, WM1 is midtrained on >50 hours of synthetic simulation data and then fine-tuned on ~2.5 hours of real data.
WM2 (Rendering Model): Maps the predicted masks to photorealistic RGB images. It is trained exclusively on real data (~2.5 hours) to learn the specific appearance of the environment.

3. Architecture Details

Backbone: Both models utilize a Stable Video Diffusion (SVD) backbone.
WM1 (Left in Fig. 2):
- Initialized from SVD and adapted via LoRA.
- Takes past/future masks and actions as input.
- Actions (6-DoF pose + 17 joint angles) are projected via MLPs and injected via frame-wise cross-attention.
- Trained on synthetic data (IsaacLab) with ground-truth masks, then fine-tuned on real data.
WM2 (Right in Fig. 2):
- Conditioned on predicted masks (from WM1), past RGB frames, and actions.
- Uses a ControlNet branch with a lightweight CNN encoder to process segmentation masks. The CNN operates in pixel space to preserve sharp boundaries that VAE encoders might blur.
- The SVD backbone is frozen; LoRA and the ControlNet/CNN are trained on real data.
- Employs classifier-free guidance via independent dropout of mask and action conditions.

4. Data Pipeline

Simulation: Generated using IsaacLab with a MimicGen-based pipeline and an exploratory sinusoidal joint generator. This covers the full 17-DoF joint range, providing diverse contact-rich motions absent in limited real datasets.
Real Data: ~2.5 hours of teleoperated demonstrations using an ORCA hand and Rokoko glove. Masks are generated automatically using SAM 3 (Segment Anything Model) as pseudo-labels, eliminating manual per-frame annotation.

Key Contributions

Segmentation as a Sim-to-Real Bridge: The authors demonstrate that using segmentation masks as an intermediate representation enables large-scale synthetic midtraining of the dynamics model, effectively bridging the sim-to-real gap.
Two-Stage World Model: The introduction of Mask2Real-WM, which separates dynamics prediction (WM1) from appearance rendering (WM2), allowing each component to be optimized for its specific domain (simulation vs. reality).
Improved Controllability: Experiments show that simulation midtraining of the dynamics model significantly improves action controllability compared to models trained only on real data.

Experimental Results

The model was evaluated on a dexterous pick-and-place benchmark using a Franka Emika Panda arm and an ORCA hand (23 DoF action space).

Action Controllability

The authors measured "command-response controllability" by perturbing individual action dimensions with sinusoidal signals and scoring the visual response.

Full Model (Sim $\to$ Real WM1 + Real WM2): Achieved a mean controllability score of 0.95 (In-Distribution) and 0.87 (Out-of-Distribution).
Real-Only WM1: Scored 0.68 (ID) and 0.51 (OOD), indicating that without synthetic midtraining, the model fails to capture fine-grained per-finger dynamics.
Monolithic Baseline (Ctrl-World): Scored significantly lower (~0.6 ID, ~0.44 OOD), capturing broad trajectories but failing to reflect fine-grained, per-dimension action effects.

Video Prediction Quality

Sharpness: The full model produces sharper predictions with better high-frequency detail compared to the monolithic baseline, which tends to produce blurry, temporally averaged outputs.
Generalization: The model maintains structural fidelity under distribution shifts (e.g., new backgrounds, absent objects) better than the baseline, though raw pixel metrics (PSNR/SSIM) can be misleading as they often favor the blurry baseline.
Ablation: Removing mask conditioning from WM2 resulted in blurry outputs, confirming that the ControlNet branch is the primary driver of spatial quality.

Significance and Claims

The paper claims that Mask2Real-WM provides a foundation for future policy evaluation in imagination by enabling long-horizon, autoregressive rollouts with high command-response controllability.

Modularity: The two-stage design offers practical modularity; WM1 can serve as a lightweight dynamics checker, and WM2 can be fine-tuned for new scenes by updating only ControlNet weights.
Scalability: The approach addresses the scarcity of real-world dexterous data by leveraging synthetic simulation for dynamics learning, a strategy the authors argue is broadly applicable where large-scale real data is unavailable.
Limitations: The authors acknowledge that the current bottleneck lies in WM1's object-state prediction (e.g., object vanishing under occlusion or duplication). They also note that the controllability scores are indicative (based on a single evaluator) and that the fixed-camera assumption limits deployment flexibility.

In summary, the paper posits that decoupling dynamics from rendering via segmentation masks, combined with simulation midtraining, is a necessary and effective strategy for achieving controllable, high-fidelity world models for dexterous manipulation.