Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

Technical Summary: Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

Problem Statement
Deploying learned manipulation policies in the real world is hindered by the prohibitive cost of collecting real robot demonstrations. While simulation offers a scalable alternative for data collection, the inherent visual and physical discrepancies between simulation and reality (the "sim-to-real gap") frequently cause transfer failures. Although "world-action models"—unified generative models that predict future visual observations alongside robot actions—have shown promise in sim-to-sim or real-to-real settings, it remains unproven whether they can successfully bridge the simulation-to-real gap for robotic manipulation without real-world fine-tuning.

Methodology
The authors propose a framework to train a world-action model entirely on synthetic data and deploy it zero-shot on a real robot. The approach integrates three core components:

1. Model Architecture (Cosmos Policy): The study utilizes Cosmos Policy, which adapts a pretrained video diffusion model (Cosmos-Predict2) for robot control. Through single-stage post-training, the model encodes actions, future observations, and value estimates as latent frames within a unified diffusion process.
2. Synthetic Data Generation:
   • Environment: Training environments are constructed using GPU-accelerated simulation with extensive domain randomization. This includes randomized textures (object surfaces, table materials, backgrounds), camera poses (perturbed wrist and third-person views), lighting conditions (varying number, position, intensity, and color temperature), and object placements.
   • Demonstrations: Expert demonstrations are generated automatically using the AnyTask motion planning pipeline (ViPR), which leverages foundation models for motion planning without human teleoperation. This process yields approximately 800 demonstrations per task across four specific tasks: lifting a banana, lifting a brick, opening a drawer, and putting a strawberry into a bowl.
3. Training and Deployment: The model is trained for 32 epochs on 40 H100 GPUs using the synthetic dataset. Crucially, the policy is deployed on a real Franka Research 3 robot without any real-world demonstrations or fine-tuning. The only real-world adaptation is the standard camera configuration (fixed RGB views from wrist and third-person cameras) used during inference.

Key Results
The study evaluates the policy's performance across four real-world tasks, comparing it against Diffusion Policy (DP) baselines trained on limited real-world data (10 or 50 demonstrations).

• Success Rates: The proposed world-action model, trained on ~800 synthetic demonstrations per task and zero real demonstrations, achieved an average success rate of 35% across the four tasks.
  • Banana Lift: 5/10
  • Brick Lift: 5/10
  • Drawer Open: 2/10
  • Strawberry Bowl: 2/10
• Comparison: This performance exceeds the DP baselines trained with 10 real demonstrations (5% average) and 50 real demonstrations (25% average). The authors note that these baselines serve as a practical reference for the cost of real data rather than a controlled architectural ablation.
• Qualitative Analysis: Visual comparisons (Figure 2) indicate that the model's predicted future frames align well with synchronized live camera observations during execution (approach, grasp, and lift stages), suggesting the model learned real-scene dynamics despite training solely in simulation.
• Generalization: The policy demonstrated out-of-distribution generalization by successfully approaching, grasping, and lifting a novel bottle object that was not part of the training set (Figure 3).

Significance and Claims
The paper claims to present, to the authors' knowledge, the first successful sim-to-real transfer of a world-action model for robotic manipulation.

The significance of this work lies in demonstrating that world-action models can inherit the primary benefit of sim-to-real learning: reducing dependence on expensive, manually collected real-world data. By achieving non-zero zero-shot manipulation performance using only automatically generated synthetic priors, the study opens a data-efficient direction for transferring generative robot policies from simulation to the real world. The authors explicitly state that this paper represents an early result of their broader work on this topic and do not claim that joint video-action prediction is categorically necessary for sim-to-real transfer, but rather that it is empirically possible to achieve transfer with this specific architecture and training regime.