🤖 AI

ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors

ExpertGen is a scalable sim-to-real framework that automates expert policy learning by initializing a frozen diffusion policy with imperfect behavior priors and refining it via reinforcement learning to achieve high success rates on complex manipulation tasks without reward engineering.

Original authors: Zifan Xu, Ran Gong, Maria Vittoria Minniti, Ahmet Salih Gundogdu, Eric Rosen, Kausik Sivakumar, Riedana Yan, Zixing Wang, Di Deng, Peter Stone, Xiaohan Zhang, Karl Schmeckpeper

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zifan Xu, Ran Gong, Maria Vittoria Minniti, Ahmet Salih Gundogdu, Eric Rosen, Kausik Sivakumar, Riedana Yan, Zixing Wang, Di Deng, Peter Stone, Xiaohan Zhang, Karl Schmeckpeper

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to do a complex task, like assembling a piece of furniture or stacking a banana on a can. Usually, you'd need to hire a human expert to show the robot exactly what to do thousands of times. But that's expensive, slow, and hard to scale.

EXPERTGEN is a new "robot teacher" system that solves this problem. It's like a smart, tireless coach that takes a few clumsy, imperfect attempts at a task and turns them into a master-level expert performance, all inside a virtual world before the robot ever touches the real world.

Here is how it works, broken down into three simple steps using a cooking analogy:

1. The "Bad Recipe" (Imperfect Behavior Priors)

Imagine you have a recipe for a cake, but it's a bit messy. Maybe it was written by a beginner, or maybe it was generated by an AI that hasn't baked much before.

The Problem: If you follow this recipe exactly, the cake might be lopsided, or it might fall apart. It's "imperfect."
The Paper's Solution: Instead of throwing the recipe away, EXPERTGEN treats this "bad recipe" as a starting point. It uses a special type of AI (called a Diffusion Policy) to learn the general "feel" of the recipe. It learns the basic motions, like "mix the batter" or "put it in the oven," even if the timing or temperature is slightly off.

2. The "Virtual Kitchen" (Massive Parallel Simulation)

Now, imagine you have a magical kitchen where you can run 1,000 simulations at the same time.

The Problem: If you just let a robot try to learn from scratch in this kitchen, it would crash into walls, drop the cake, and never figure out how to bake. It needs a guide.
The Paper's Solution: This is where the magic happens. EXPERTGEN takes that "bad recipe" (the imperfect prior) and puts it in the virtual kitchen. It uses a technique called Diffusion Steering.
- The Analogy: Think of the robot's movements as a river flowing down a hill. The "bad recipe" sets the general direction of the river (it knows it needs to flow toward the ocean/task goal). However, the river might be too wide or hit some rocks.
- The Fix: EXPERTGEN acts like a dam operator. It doesn't change the river's path entirely (which would make the robot forget the human-like style). Instead, it just nudges the water slightly at the very beginning of the flow to steer it around the rocks and toward the goal.
- The Result: The robot learns to succeed by making tiny adjustments to its starting point, keeping the "human-like" style but fixing the mistakes. It does this millions of times in the virtual kitchen until it becomes a master chef.

3. The "Real World Transfer" (Sim-to-Real)

Now the robot is a master chef in the virtual kitchen. But can it cook in a real kitchen with real lighting, real dust, and real wobbly tables?

The Problem: Virtual worlds are perfect; real worlds are messy. A robot trained in a perfect simulation often fails when it sees a real object.
The Paper's Solution: EXPERTGEN uses a technique called DAgger (which sounds like a sword, but is actually a teaching method).
- The Analogy: Imagine the virtual robot (the Master Chef) is teaching a student robot (the Student) who only has a camera and no access to the "secret state" of the kitchen.
- The Process: The student tries to cook. When it gets confused or makes a mistake, the Master Chef steps in and says, "No, do it this way." The student records this correction. They repeat this over and over.
- The Result: The student learns to handle the messy, real-world lighting and camera angles by constantly checking with the Master Chef. Eventually, the student becomes so good that it can cook perfectly on its own, even if the lighting changes or the table shakes.

Why is this a big deal?

No "Reward Engineering" Headaches: Usually, teaching a robot requires a human to write complex rules like "If the robot drops the cup, give it -1 point." This is hard to get right. EXPERTGEN just says, "Did you finish the task? Yes? Good. No? Try again." It figures out the rest on its own.
It Learns from Failure: Most robots give up when they fail. EXPERTGEN specifically learns how to recover from mistakes. If the robot knocks the banana over, it learns how to pick it back up and stack it, rather than just stopping.
It's Cheap and Fast: You don't need to hire humans to record thousands of hours of video. You can generate the "imperfect" data with a computer script or a Large Language Model (LLM), and then let the simulation do the heavy lifting.

The Bottom Line

EXPERTGEN is like taking a rough draft of a robot's behavior, running it through a super-fast, infinite practice loop where it learns to fix its own mistakes, and then teaching a camera-based student how to copy that perfection in the real world.

The result? Robots that can handle messy, real-world tasks (like industrial assembly or picking up fruit) with a 90%+ success rate, even when they start with very little data and no human experts to guide them step-by-step.

1. Problem Statement

The core challenge addressed by this paper is the scarcity of high-quality, expert-level robotics data required to train generalizable and robust policies for real-world deployment.

The Bottleneck: While Vision-Language-Action (VLA) models show strong semantic understanding, translating this to reliable physical execution is hindered by the lack of closed-loop, temporally aligned perception-action sequences. Collecting real-world expert data via teleoperation is prohibitively expensive and slow.
Limitations of Existing Sim-to-Real Approaches:
- Scripted Policies/LLM Generation: Often lack behavioral diversity, fail to cover failure states, and suffer from dynamics mismatches when transferred to real hardware.
- Standard Reinforcement Learning (RL): Scaling RL to generate expert data is difficult because it typically requires carefully engineered reward functions (reward shaping), which demands significant domain expertise and limits scalability across diverse tasks.
- Imperfect Priors: Existing methods struggle to refine "imperfect" demonstrations (e.g., those with incomplete state coverage or no failure recovery) into robust expert policies without losing the safety guarantees of the original data.

2. Methodology: The EXPERTGEN Framework

EXPERTGEN is an end-to-end framework designed to transform a small set of imperfect demonstrations into high-quality, sim-to-real-ready expert policies. It operates in three distinct phases:

Phase 1: Generative Behavior Prior Modeling

Input: A small dataset of "imperfect" demonstrations. These can be synthesized by Large Language Models (LLMs), scripted policies, or human teleoperation. They may exhibit limited state coverage, lack failure recovery, or have dynamics mismatches.
Model: A state-based Diffusion Policy is trained on these imperfect demonstrations.
Goal: To create a "behavior prior" ( $\pi_P$ ) that captures feasible, human-like motion manifolds but is not yet optimized for task success. This prior acts as a safety regularizer, ensuring the robot stays within physically feasible and safe behavior spaces.

Phase 2: Expert Policy Acquisition via Diffusion Steering RL

Mechanism: The framework employs Diffusion Steering Reinforcement Learning (DSRL).
- Instead of fine-tuning the weights of the pretrained diffusion model (which could destroy the learned motion manifold), DSRL learns a lightweight steering policy that optimizes the initial noise ( $z_0$ ) fed into the diffusion model.
- The diffusion model itself remains frozen.
Algorithm: The steering policy is trained using FastTD3, a scalable, off-policy RL algorithm designed for massively parallel simulation (e.g., IsaacLab).
Reward: The system uses sparse, task-level rewards (binary success/failure) with no reward engineering.
Outcome: The RL agent learns to steer the diffusion model toward high task success rates while preserving the natural, safe motion characteristics of the original prior. This enables robust failure recovery and generalization to unseen initial configurations.

Phase 3: Visuomotor Policy Distillation (DAgger)

Challenge: The expert policy learned in Phase 2 relies on privileged state information (e.g., exact object poses) unavailable on real robots.
Solution: A DAgger (Dataset Aggregation) distillation process is used.
- The "Teacher" (state-based expert) runs in simulation to generate trajectories.
- The "Student" (visuomotor policy) observes only raw inputs (RGB images/Point clouds).
- The Teacher provides corrective actions for states visited by the Student, creating a dataset that covers the Student's distribution.
Sim-to-Real: Extensive visual domain randomization (textures, lighting, camera poses) is applied during distillation to ensure the final policy generalizes to the real world without further fine-tuning (zero-shot transfer).

3. Key Contributions

Scalable Diffusion Steering: Demonstrated that steering the initial noise of a diffusion model in massively parallel simulation preserves the natural motion manifold while significantly boosting success rates under sparse rewards.
End-to-End Framework (EXPERTGEN): Introduced a pipeline that converts a handful of imperfect demonstrations (even LLM-generated or scripted) into robust expert policies without manual reward engineering.
Robust Zero-Shot Transfer: Validated that policies distilled via DAgger from these simulated experts can be deployed on real hardware with high success rates, overcoming the sim-to-real gap.
Performance on Sparse Rewards: Proved that combining diffusion priors with FastTD3 allows for effective learning in sparse reward regimes where standard RL often fails.

4. Experimental Results

The framework was evaluated on two major benchmarks: ANYTASK (general manipulation) and AutoMate (industrial assembly).

Success Rates:
- ANYTASK: EXPERTGEN achieved the highest success rates across all 8 tasks, reaching ~100% on short-horizon tasks (e.g., Lift, Open Drawer) and 85% on long-horizon tasks, outperforming baselines like Residual RL, Diffusion Policy, and PPO-based variants.
- AutoMate: Achieved a 90.5% overall success rate on high-precision peg-insertion tasks, significantly outperforming all baselines.
Failure Recovery: Under perturbations (random gripper opening, external forces), EXPERTGEN showed minimal performance degradation (e.g., only 0.5% drop in some cases), whereas base diffusion policies failed completely (success rate dropped to near zero).
Data Efficiency: The system requires only a small number of imperfect demonstrations (approx. 200) to initialize a prior that can be effectively refined. Performance plateaus after this threshold.
Real-World Deployment:
- Deployed on a Franka Emika robot with a Robotiq gripper.
- Point-Cloud Policy: Achieved 75-85% success on tasks like "Lift Banana" and "Push Pear," outperforming standard Behavioral Cloning (BC) which failed on complex articulation tasks.
- RGB Policy: Achieved 80% success on the "Lift Banana" task in zero-shot trials, demonstrating robustness to visual distractors and clutter.

5. Significance

EXPERTGEN addresses a critical bottleneck in robotics: the transition from semantic understanding to reliable physical execution.

Democratization of Data: It removes the dependency on massive, high-quality human datasets or complex reward engineering, allowing robots to learn from "imperfect" or synthetic data.
Safety and Generalization: By keeping the diffusion prior frozen and only steering the noise, the method ensures that the robot explores safely within human-like behavior manifolds, preventing the "catastrophic forgetting" or unsafe exploration often seen in pure RL.
Scalability: The use of massively parallel simulation and off-policy learning (FastTD3) makes the approach scalable to diverse tasks and long-horizon problems.
Practical Impact: The successful zero-shot transfer to real hardware demonstrates a viable path toward deploying generalist robot policies in unstructured real-world environments.