💻 computer science

ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation

The paper proposes ComSim, a novel hybrid approach that combines classical and neural simulation within a closed-loop real-sim-real pipeline to generate scalable, high-quality action-video datasets that significantly reduce the sim2real domain gap and improve real-world robot policy performance.

Original authors: Yiran Qin, Jiahua Ma, Li Kang, Wenzhan Li, Yihang Jiao, Xin Wen, Xiufeng Song, Heng Zhou, Jiwen Yu, Zhenfei Yin, Xihui Liu, Philip Torr, Yilun Du, Ruimao Zhang

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiran Qin, Jiahua Ma, Li Kang, Wenzhan Li, Yihang Jiao, Xin Wen, Xiufeng Song, Heng Zhou, Jiwen Yu, Zhenfei Yin, Xihui Liu, Philip Torr, Yilun Du, Ruimao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to do chores, like stacking blocks or shaking a bottle. In the past, you had two main options, and both had big problems:

The "Real Life" Method: You physically move the robot's arms thousands of times to show it what to do.
- The Problem: This is like trying to teach a child to swim by throwing them in the ocean every day. It's slow, expensive, and you can't cover every possible wave or current. You run out of time and money.
The "Video Game" Method: You train the robot inside a perfect computer simulation (like a high-end video game).
- The Problem: This is like teaching someone to drive using a flight simulator. The physics look okay, but the tires don't grip the road the same way, and the lighting is too perfect. When the robot tries to drive a real car, it crashes because the "game" didn't feel like the "real world."

The "Fake It 'Til You Make It" Method (Neural Simulators):
Recently, scientists tried using AI to turn those "game" videos into "real" videos.

The Problem: The AI is a bit of a daydreamer. It might make the robot's hand look real, but it forgets how the hand actually moves. The robot might try to grab a cup, but in the video, the cup magically teleports. The robot learns the wrong lessons.

The Solution: "Compositional Simulation" (The Best of Both Worlds)

This paper introduces a clever new recipe called Compositional Simulation. Think of it as a Master Chef who combines the precision of a recipe book with the taste of a real meal.

Here is how it works, step-by-step:

1. The "Digital Twin" Setup

First, the researchers build a computer simulation that is a perfect mirror of their real robot lab. They match the colors of the table, the size of the blocks, and the angle of the camera. It's like setting up a movie set that looks exactly like the real kitchen.

2. The "Rehearsal" (Real-to-Sim)

They take a human, put them in the real lab, and have them do a task (like shaking a bottle). They record the video.
Then, they take that exact same movement and replay it inside the computer simulation.

Result: Now they have a pair of videos: one from the real world and one from the game, showing the exact same action.

3. The "Translator" (The Neural Simulator)

They train a special AI (the Neural Simulator) using these pairs. This AI learns to look at the "game" video and say, "Okay, I know how this looks in the game, but I need to make it look like the real video."

The Magic Trick: Unlike other AIs that just guess, this one is forced to keep the actions exactly the same. It can change the lighting and the texture of the table, but it cannot change the fact that the robot is shaking the bottle. It's like a translator who changes the accent and vocabulary but never changes the meaning of the sentence.

4. The "Mass Production" (Sim-to-Real)

Now comes the scaling part. The computer simulation can generate millions of different scenarios instantly (different colored blocks, different table positions, different speeds).

The researchers let the simulation run wild, creating thousands of "game" videos.
They feed these into their "Translator" AI.
Result: The AI spits out thousands of videos that look photorealistic (like they were filmed in a real lab) but are actually generated by the computer.

Why This is a Big Deal

Imagine you are teaching a dog to fetch.

Old Way: You throw a ball 10 times in the park. The dog learns a little.
New Way: You use this "Compositional Simulation" to generate 10,000 videos of the dog fetching balls in the park, in the rain, on grass, on sand, and with different colored balls. You show these to the dog (or the robot).
The Outcome: When you finally take the robot out to the real park, it's not surprised. It's seen every possible variation of the task already.

The Results

The paper tested this by training robots to do tasks like stacking blocks and moving cards.

Robots trained only on real data (few examples) failed often.
Robots trained on "fake" game data failed because the physics were wrong.
Robots trained with this new method were like superheroes. They succeeded at much higher rates and could handle new objects and new table layouts they had never seen before.

In short: This paper figured out how to use a computer to generate infinite, perfect "practice runs" that feel exactly like the real world, allowing robots to learn faster, cheaper, and better than ever before.

1. Problem Statement

Robotics faces a critical bottleneck in acquiring large-scale, high-quality training data. While foundational models (LLMs, world models) have advanced robot capabilities, training them requires vast amounts of action-video pairs.

Real-world collection: Is expensive, labor-intensive, and limited in diversity, failing to cover the full spectrum of real-world scenarios.
Classical Simulation (e.g., MuJoCo, Isaac Lab): Can generate massive datasets with precise action labels but suffers from a significant Sim2Real domain gap (differences in appearance, lighting, texture, and physics), leading to poor policy transfer.
Neural Simulators (Video Generation Models): Can generate realistic-looking videos but often suffer from hallucinations, lack of 3D consistency, and, crucially, weak action-video alignment (the generated video does not faithfully follow the input action trajectory).

The core challenge is to generate scalable, diverse training data that maintains real-world visual fidelity while ensuring strict action consistency.

2. Methodology: Compositional Simulation

The authors propose Compositional Simulation, a hybrid framework that combines the strengths of classical simulation (action precision) and neural simulation (visual realism) via a Real-Sim-Real closed-loop pipeline.

A. The Real-Sim-Real Pipeline

The process involves three main stages:

Real2Sim Alignment (Data Collection):
- A small set of real-world trajectories is collected via teleoperation.
- A "Digital Twin" simulation environment is constructed with strict alignment to the real world:
  - Background/Object Alignment: Matching colors, sizes, and textures of assets.
  - Camera Calibration: Aligning intrinsic and extrinsic camera parameters.
  - Object Localization: Precisely transferring object positions from the real scene to the simulator.
- The same real-world actions are replayed in the simulator to generate paired data $(V_{sim}, V_{real}, A)$ .
Neural Simulator Training (Sim2Real):
- A Neural Simulator is trained to transform classical simulation videos ( $V_{sim}$ ) into real-world representations ( $V_{real}$ ) while preserving the action sequence ( $A$ ).
- Compositional Dynamic Video Generation: The model (based on a Diffusion Transformer, DiT) is conditioned on two distinct dynamics:
  - Control Dynamics: The action trajectory ( $A$ ).
  - Visual Dynamics: The simulated observation ( $V_{sim}$ ).
- During sampling, scores from both dynamics are composited to guide the generation. This ensures the output video is visually realistic (driven by visual dynamics) but strictly adheres to the input action (driven by control dynamics).
- Loss Function: Minimizes the discrepancy between the generated video and the real-world video while maintaining action alignment.
Scalable Data Generation:
- Using a rule-based simulator (RoboTwin), a massive dataset of diverse trajectories is generated using action primitives (e.g., grasp, push, stack) orchestrated by an LLM (GPT-5).
- These simulated trajectories are passed through the trained Neural Simulator to produce "Pseudo Real Data"—large-scale $(V_{real}, A)$ pairs that look real but are generated synthetically.

B. Policy Training

The final robot policy is trained via Joint Training on a mixture of:

A small amount of Real Data (collected from the actual environment).
A large amount of Pseudo Real Data (generated via the Compositional Simulation pipeline).

3. Key Contributions

Concept of Compositional Simulation: A novel paradigm that bridges the gap between classical and neural simulation by decoupling action control from visual generation, then recombining them to ensure both physical consistency and visual realism.
Real-Sim-Real Data Pipeline: A robust framework for creating a "Digital Twin" environment and training a neural simulator that transforms low-fidelity simulation data into high-fidelity, action-consistent real-world data.
Compositional Dynamic Video Generation: A technical innovation using a DiT conditioned on both control and visual dynamics to prevent hallucinations and ensure precise action-video alignment.
Scalable Solution: Demonstrates a method to generate large-scale, diverse datasets covering broad spatial and object distributions using minimal real-world data.

4. Experimental Results

The authors evaluated their approach on 8 robotic manipulation tasks (e.g., Shake Bottle, Stack Blocks, Move Playing Card) using a Diffusion Policy (DP) backbone.

Video Quality (Sim2Real Transfer):
- The full pipeline (Ours-Full) achieved state-of-the-art performance in perceptual metrics (PSNR, SSIM, CLIP Score) and distributional metrics (FID, FVD), significantly outperforming baselines like Stable Diffusion, Zero-Shot models, and variants using only control or visual conditioning.
- Qualitative results showed the method successfully preserved object identity, background consistency, and precise gripper movements, whereas baselines suffered from hallucinations or action drift.
Real-World Policy Performance:
- Success Rates: Policies trained with 10 Real + 200 Pseudo Real data significantly outperformed those trained with only real data (10 Real) or traditional simulation data (10 Real + 200 Sim).
- Generalization:
  - Spatial: The method enabled strong generalization to Out-of-Distribution (OOD) spatial layouts (objects placed in unseen regions), where pure real-data training failed completely.
  - Object: The method generalized to new object instances (e.g., different bottle brands, different card colors) not seen during real-world data collection, whereas traditional simulation failed to transfer this generalization.
- Zero-Shot Potential: Even without any real-world data (200 Pseudo Real only), the policy achieved non-trivial success rates, demonstrating the high fidelity of the generated data.

5. Significance

This work addresses the fundamental data scarcity problem in embodied AI. By proving that a small amount of real-world data can be leveraged to "style-transfer" massive amounts of classical simulation data into a realistic domain, the paper offers a scalable, cost-effective path to training robust robot policies. It effectively bridges the Sim2Real gap, enabling robots to learn from diverse, complex scenarios that would be impossible to collect manually, while maintaining the critical link between action and observation required for reliable control.