💻 computer science

Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation

This paper introduces Simulation Distillation (SimDist), a framework that distills structural priors from simulators into latent world models to enable rapid, stable, and data-efficient real-world adaptation in robotics by transferring reward and value models directly from simulation, thereby avoiding the challenges of long-horizon credit assignment and exploration in low-data regimes.

Original authors: Jacob Levy, Tyler Westenbroek, Kevin Huang, Fernando Palafox, Patrick Yin, Shayegan Omidshafiei, Dong-Ki Kim, Abhishek Gupta, David Fridovich-Keil

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jacob Levy, Tyler Westenbroek, Kevin Huang, Fernando Palafox, Patrick Yin, Shayegan Omidshafiei, Dong-Ki Kim, Abhishek Gupta, David Fridovich-Keil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to walk across a room or pick up a delicate object. You have two choices:

The "Real World" Method: Let the robot try, fail, fall over, break things, and learn slowly. This is dangerous, expensive, and takes forever.
The "Simulation" Method: Let the robot practice in a video game (a simulator) where it can try a million times in a minute without breaking anything.

The Problem:
The video game isn't perfect. The physics are slightly different from reality. A robot that learns to walk perfectly in the game often trips and falls the moment it steps onto the real floor. This is called the "Sim-to-Real Gap."

Most current methods try to fix this by letting the robot relearn everything from scratch on the real floor, which is slow and unstable.

The Solution: SimDist (Simulation Distillation)
The paper introduces a new framework called SimDist. Think of it as a Master Chef and a Sous-Chef relationship.

The Analogy: The Master Chef and the Sous-Chef

Imagine you want to cook a perfect meal (the robot's task) in a new, unfamiliar kitchen (the real world).

1. The Master Chef (The Simulator):
In the video game, you have a Master Chef who knows the recipe perfectly. They know exactly how long to boil the pasta, how much salt to add, and what the final dish should taste like.

What SimDist does: It doesn't just let the Master Chef cook. It forces the Master Chef to cook badly on purpose sometimes (adding too much salt, burning the toast) and then fix the mistakes. This teaches the system what "failure" looks like and how to recover.
The Result: The Master Chef creates a massive library of knowledge: "Here is what success looks like, here is what failure looks like, and here is how to fix it."

2. The Sous-Chef (The World Model):
Instead of teaching the robot the whole recipe (which includes the specific taste of the Master Chef's kitchen), SimDist teaches the robot a World Model.

Think of the World Model as the robot's internal map and compass.
It learns the structure of the task: "If I move my arm this way, the object moves that way."
Crucially, it learns what success feels like (the Reward) and how close it is to winning (the Value).

3. The Transfer (The "Distillation"):
When the robot moves to the real kitchen, SimDist does something clever:

It freezes the "Map," the "Compass," and the "Taste Test" (the Reward and Value models). These are universal truths that don't change between the video game and reality.
It only updates the "Muscle Memory" (the Dynamics). This is the part that says, "In this specific real kitchen, the floor is slippery, so I need to take smaller steps."

How It Works in Practice

Pre-training (The Simulation): The robot spends hours in the video game. It learns to recognize objects, knows what a "win" looks like, and learns the general rules of physics. It also practices recovering from mistakes.
Deployment (The Real World): The robot is dropped into the real world.
- It uses its frozen knowledge to plan: "I know I need to grab that peg. I know if I miss, I lose points."
- It uses a tiny bit of real-world data (just 15–30 minutes!) to update its muscle memory. It learns, "Oh, the real floor is slippery. I need to adjust my foot placement."
The Magic: Because the robot already knows what to do and why it's doing it (from the simulation), it doesn't have to relearn the whole task. It just tweaks its movements to fit the new environment.

Why Is This a Big Deal?

Speed: Other methods might need hours or days of trial-and-error in the real world. SimDist gets it working in 15 to 30 minutes.
Stability: Other methods often "forget" what they learned in the simulator when they start learning in the real world (like a student forgetting math when they start learning history). SimDist keeps the important math (the rules of the task) and only changes the history (the specific environment).
Safety: Since the robot already knows the "rules of the game" from the simulator, it doesn't need to crash into things to learn.

Summary

SimDist is like giving a robot a perfect textbook (from the simulator) and a compass (the reward model), then letting it spend just a few minutes in the real world to learn the local terrain (the dynamics). It allows robots to go from "Video Game Pro" to "Real World Expert" almost instantly, without breaking anything or needing a massive amount of data.

1. Problem Statement

The central challenge addressed is the Sim-to-Real gap in robotics. While simulators allow for scalable data generation and training, the inevitable mismatch between simulated and real-world dynamics often causes policies trained in simulation to fail when deployed on physical robots.

Existing approaches face specific limitations:

End-to-End Policy Finetuning: Directly finetuning a policy on limited real-world data often leads to catastrophic forgetting of pre-trained priors. It forces the agent to relearn the entire task structure (representations, rewards, and dynamics) simultaneously, which is unstable in low-data regimes.
Credit Assignment: Reinforcement Learning (RL) in the real world struggles with long-horizon credit assignment and exploration when data is scarce.
Model-Based RL (MBRL): Standard online MBRL methods are computationally intensive and often fail to generalize because they attempt to bootstrap behavior from scratch without leveraging rich simulation priors.

2. Methodology: Simulation Distillation (SimDist)

The authors propose SimDist, a framework that decouples the learning of global task structure from local environment dynamics. The core insight is that while low-level dynamics (friction, contact physics) differ between simulation and reality, the high-level semantic structure (object locations, goal states, reward logic) remains largely invariant.

SimDist operates in three main phases:

A. Pretraining in Simulation (Distillation)

Instead of training a policy directly, SimDist trains a Planning-Oriented Latent World Model using massive parallel simulation.

Data Generation: A privileged state-based expert policy (trained via PPO) and intermediate checkpoints are used to generate a massive dataset. Crucially, the authors inject action perturbations and mix sub-optimal policies to create diverse trajectories, including failures and recoveries. This ensures the world model learns to reason about counterfactual trajectories, not just expert paths.
World Model Architecture: The model consists of:
- Encoders: Map raw observations (images, proprioception) to a latent state $z_t$ .
- Latent Dynamics: Predicts future latent states given actions.
- Reward & Value Heads: Predict future rewards and values directly from the latent trajectory.
- Base Policy: Predicts action chunks to warm-start planning.
Loss Function: The model is trained to minimize errors in latent dynamics, reward prediction, value prediction, and behavior cloning (for expert actions).

B. Real-World Adaptation (System Identification)

Upon deployment, the framework freezes the components learned in simulation that represent global structure:

Frozen: State Encoders, Reward Models, and Value Functions.
Adaptable: Only the Latent Dynamics Model is finetuned.
Process: The robot collects short real-world trajectories (15–30 minutes). The dynamics model is updated via supervised system identification to minimize the prediction error between the model's latent state and the encoder's output of real observations.
Planning: The robot uses Model Predictive Path Integral (MPPI) control. It samples action sequences, evaluates them using the frozen reward/value models and the adapted dynamics model, and executes the optimal trajectory.

C. Iterative Improvement

The process is iterative: collect real data $\rightarrow$ finetune dynamics $\rightarrow$ plan with updated model $\rightarrow$ collect more data. Because the reward and value functions are already accurate (transferred from sim), the planner immediately improves as the dynamics model becomes more accurate, avoiding the need for difficult reward learning in the real world.

3. Key Contributions

SimDist Framework: A novel paradigm that reduces real-world adaptation to a simple, short-horizon supervised learning problem (dynamics finetuning) by freezing global task priors. This avoids long-horizon credit assignment issues.
Zero-Shot Transfer of Reward/Value: Demonstrates that reward and value models trained on privileged simulation states can transfer zero-shot to the real world using only raw perception, enabling dense planning signals without real-world reward engineering.
Data Efficiency & Stability: Achieves substantial performance gains on complex tasks with only 15–30 minutes of real-world interaction, significantly outperforming standard RL baselines which often collapse or fail to improve.
Scalable Pretraining: Leverages massive parallel simulation with perturbed expert data to create a robust world model capable of reasoning about off-policy trajectories.

4. Experimental Results

The authors evaluated SimDist on four tasks: two precise manipulation tasks (Peg Insertion, Table Leg) and two quadruped locomotion tasks (Slippery Slope, Foam).

Performance: SimDist achieved ~2x higher success rates compared to state-of-the-art baselines (RLPD, IQL, Diffusion Policy, SGFT).
Sample Efficiency: While baselines often showed catastrophic forgetting or stagnation, SimDist showed monotonic improvement with minimal data (15–30 mins).
Robustness: In the "Peg Insertion" task, SimDist handled a "Wide" initial condition range (35cm x 35cm) significantly better than Diffusion Policy, which failed on narrow ranges.
Ablation Studies:
- Data Diversity: Training only on expert data (no perturbations) caused a sharp performance drop, proving the necessity of diverse failure/recovery data for planning.
- Architecture: Using Transformer-based sequence-to-sequence heads for rewards/values outperformed per-timestep MLPs.
- Reconstruction Loss: Adding raw image reconstruction loss hurt manipulation performance, suggesting that task-irrelevant visual details interfere with compact latent dynamics.

5. Significance

This work represents a significant shift in Sim-to-Real strategies. Instead of treating the real world as a place to relearn everything from scratch, SimDist treats it as a place to calibrate the physics engine (dynamics) while retaining the "brain" (reward, value, and representation) learned in simulation.

Practical Impact: It enables robots to adapt to new physical environments (e.g., different friction, unmodeled contacts like foam) in minutes rather than hours or days, making autonomous deployment in unstructured environments more feasible.
Theoretical Insight: It validates the hypothesis that global task structure is invariant across domains, while local dynamics are not. By decoupling these, the method sidesteps the instability of off-policy RL and the data inefficiency of online model-based RL.
Future Direction: The approach suggests a path toward "World Model as a Service," where a single pre-trained world model can be rapidly adapted to diverse real-world scenarios via lightweight system identification.