💻 computer science

GPU-Parallel Multi-Task Reinforcement Learning with Demonstration Guided Policy Optimization

This paper introduces MT-Libero, a GPU-parallel multi-task reinforcement learning benchmark for structured manipulation tasks, and proposes DGPO, an on-policy demonstration-guided method that combines importance-weighted PPO with adaptive behavior cloning to effectively train heterogeneous task suites under sparse rewards.

Original authors: Rui Zhang, Qiwei Wu, Zhengyu Zhang, Tao Li, Yunrong Guo, Junjie Lai, Renjing Xu, Weihua Zhang

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rui Zhang, Qiwei Wu, Zhengyu Zhang, Tao Li, Yunrong Guo, Junjie Lai, Renjing Xu, Weihua Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot arm to do chores. In the past, if you wanted the robot to learn how to pour coffee, you would train it for coffee. Then, if you wanted it to learn how to open a drawer, you had to wipe its memory and start a completely new training session just for the drawer. It was like hiring a different specialist for every single job.

This paper introduces a new way to train robots that is faster, smarter, and more versatile. It does two main things: it builds a massive, high-speed training gym, and it invents a new teaching method that uses human "demonstrations" as a guide without forcing the robot to just copy us blindly.

Here is a breakdown of their approach using simple analogies:

1. The Problem: The "One-Task-at-a-Time" Bottleneck

Think of current robot training like a single-lane road. You can only send one car (one robot learning one task) at a time. Even though we have powerful computers (GPUs) that could handle thousands of cars, we are stuck sending them one by one. This is slow and inefficient.

2. The Solution Part 1: MT-Libero (The "Super Gym")

The authors built MT-Libero, which is like turning that single-lane road into a massive, multi-lane superhighway.

The Analogy: Imagine a giant gym where 40 different types of robots are training simultaneously in the same room. Instead of building 40 separate gyms, they built one giant, shared gym where every robot has its own station but they all share the same equipment, the same coach, and the same schedule.
How it works: They took a standard set of robot tasks (like stacking blocks, opening drawers, or moving objects) and programmed the computer to run all of them at the exact same time on a single graphics card.
The Result: They can train a robot on 40 different tasks at once, 1,600 times faster than before, using a tiny fraction of the computer memory. It's like training a "generalist" robot that knows a little bit about everything, rather than a "specialist" that only knows one thing.

3. The Solution Part 2: DGPO (The "Smart Mentor")

Training a robot on 40 tasks at once is hard because some tasks are easy (like picking up a light block) and some are hard (like opening a sticky drawer). If the robot gets good at the easy ones, it might stop trying to learn the hard ones. Also, sometimes the robot gets stuck and doesn't know what to do.

This is where DGPO comes in. Think of it as a Smart Mentor who watches a human expert perform the tasks.

The Old Way: Some methods just say, "Copy the human exactly." If the human makes a mistake, the robot copies the mistake. Other methods say, "Ignore the human and figure it out yourself," which takes forever.
The DGPO Way: The mentor says, "I'll watch the human to get you started, but I'll let you figure out the rest."
- When the robot is struggling: The mentor leans in close and says, "Hey, look at what the human did here. Do that." (This is called Adaptive Behavior Cloning).
- When the robot is doing well: The mentor steps back and says, "Great job! Now try to improve on your own." (This is the standard Reinforcement Learning part).
- The "Fairness" Rule: The mentor also keeps a scoreboard. If the robot is failing at the "hard" tasks but acing the "easy" ones, the mentor forces the robot to spend more time practicing the hard ones. This ensures the robot gets good at everything, not just the easy stuff.

4. The Results: A "VLA-like" Robot

The paper tested this system on a variety of tasks (Goal, Object, Spatial, and Long-horizon tasks).

The Outcome: The robot trained with MT-Libero and DGPO became a true "generalist." It learned to do all 40 tasks in a single training session.
Comparison: It beat robots trained without human help (which were slow) and robots trained by just copying humans (which were rigid and couldn't improve).
Real-World Check: They even took a robot trained in this computer simulation and tried it on a real robot arm in a real room. It worked surprisingly well, showing that the skills learned in the "Super Gym" could transfer to the real world.

Summary

In short, this paper says:

Stop training robots one by one. Build a shared, high-speed environment where they learn many tasks together (MT-Libero).
Don't just copy humans; learn from them. Use human demonstrations as a flexible guide that helps when the robot is stuck but lets the robot improve on its own when it's ready (DGPO).

The result is a robot that learns faster, handles a wider variety of jobs, and gets better at the hard stuff without needing to be retrained from scratch for every new chore.

Technical Summary: GPU-Parallel Multi-Task Reinforcement Learning with Demonstration Guided Policy Optimization

Problem Statement

While large-scale GPU-parallel reinforcement learning (RL) has transformed robot simulation into a high-throughput training substrate, current systems predominantly optimize "specialist" policies—one objective, one policy, one training run at a time. This approach fails to leverage the potential for a single policy to acquire a breadth of structured manipulation skills. The transition to Multi-Task Reinforcement Learning (MTRL) in massively parallel settings introduces new algorithmic challenges beyond sample collection, specifically regarding value stability, task imbalance, and the effective utilization of prior demonstration data. Furthermore, existing benchmarks like LIBERO, while rich in structured task variations (object, spatial, goal, and long-horizon), were originally designed for CPU-based imitation learning and lack the heterogeneous, GPU-parallel execution substrate required to train a single generalist policy across diverse task families.

Methodology

The paper proposes a two-part solution: a benchmark construction methodology (MT-Libero) and a novel optimization algorithm (DGPO).

1. MT-Libero: A GPU-Parallel Benchmark Construction

MT-Libero is an instantiation of a scalable construction methodology using Isaac Lab and LIBERO assets. It addresses the gap between structured task definitions and GPU-parallel execution by:

Separating Semantics from Execution: Task descriptors, reusable assets (converted to USD), reset rules, success predicates, and reward interfaces are compiled into a single vectorized training loop.
Heterogeneous Parallelism: Instead of launching independent simulators per task, MT-Libero executes heterogeneous task groups (Goal, Object, Spatial, Long) within one vectorized environment. This allows a shared simulator, renderer, rollout buffer, and policy update stack to handle 40 distinct tasks simultaneously.
Input Flexibility: The framework supports both state-input policies (using task embeddings and proprioceptive data) and visual-input policies (using frozen ViT encoders for third-person and wrist camera images).
Curriculum and Randomization: It incorporates physical randomization and a gripper curriculum that initially follows demonstration commands before transitioning to learned control.

2. DGPO: Demonstration-Guided Policy Optimization

To address sparse success signals and limited prior data, the authors introduce DGPO, an on-policy method that combines PPO with adaptive behavior cloning. Unlike methods that treat demonstrations as fixed imitation targets or offline replay data, DGPO uses them as task-aware guidance while preserving online improvement.

Importance-Weighted PPO (IW-PPO): To manage task imbalance, DGPO reallocates the PPO gradient budget based on per-task success rates. It uses an Exponential Moving Average (EMA) of success rates to assign weights ( $w_t$ ). Tasks performing below the multi-task mean receive higher weights, while easier tasks receive lower weights. This avoids the pitfalls of loss-based weighting (e.g., FAMO), which can be confounded by reward scales or critic fit in on-policy RL.
Adaptive Behavior Cloning (ABC): DGPO adds a regularization term to the actor that penalizes deviation from matched demonstration actions ( $a^\star_t$ $a_{t}^{⋆}$ ) for the current rollout state ( $o_t$ $o_{t}$ ). Crucially, the strength of this imitation ( $\beta_k$ $β_{k}$ ) is adaptive:
- It is determined by the task's current success rate ( $\tau_k$ ).
- "Weak" students (low success) receive strong demonstration pressure ( $\beta \approx \beta_{max}$ ).
- "Mastered" tasks (high success) receive minimal pressure ( $\beta \approx \beta_{min}$ ).
- This allows the policy to specialize using demonstrations where beneficial but continue to optimize via online RL as competence grows.
Unified Objective: The total loss combines the weighted PPO objective (policy, value, and entropy) with the adaptive BC term, ensuring the policy remains on-policy while being guided by demonstrations.

Key Contributions

MT-Libero Benchmark: The authors propose and instantiate a GPU-parallel construction methodology for structured multi-task robot RL. It enables the training of generalist policies on heterogeneous task suites (40 tasks) in a single loop, supporting both state and visual inputs, varied rewards, and curricula.
DGPO Algorithm: They introduce a demonstration-guided PPO method that integrates cross-task importance weighting with adaptive behavior cloning. This approach allows prior data to guide specialization without sacrificing the stability and online improvement benefits of on-policy PPO.
Empirical Validation: The paper demonstrates that DGPO outperforms prior-free RL and existing demonstration-based baselines (including MT-RLPD, MT-RFCL, MT-DAPG, and MT-DeepMimic) in both state and visual settings, achieving higher mean success rates and better task distribution coverage.

Experimental Results

The experiments evaluate the construction recipe, the efficacy of DGPO, and the trade-offs of demonstration pressure.

Benchmark Performance: MT-Libero achieves 3,825 steps per second (SPS) on a single L20 GPU with 1,600 parallel environments, significantly outperforming a MuJoCo CPU baseline (2,274 SPS) while using far less memory (8.2 GB VRAM vs. 2.3 TB RAM). End-to-end PPO training scales to ~26,800 SPS on 8 GPUs.
Method Comparison:
- State-Input: DGPO achieves a mean success rate of 85.2% across all suites, significantly outperforming the next best baseline (MT-PPO at 42.5%). The advantage is particularly pronounced in the "Long" (long-horizon) suite (59.1% vs. 19.8%).
- Visual-Input: DGPO achieves 69.8% mean success, again outperforming all baselines. Notably, off-policy baselines (MT-RLPD, MT-RFCL) failed to converge in the visual setting under the same memory constraints due to replay buffer limitations.
Ablation Studies:
- Removing IW-PPO significantly degrades performance on difficult "tail" tasks (20th percentile success drops from 4.3% to 1.9%), confirming its role in balancing gradient allocation.
- Removing Adaptive BC causes a catastrophic collapse in learning (mean success drops to 5.1%), highlighting the necessity of demonstration guidance for initial convergence.
Generalization: DGPO shows a small trade-off in generalization within the demonstration range compared to less specialized policies, but this is outweighed by its gains in nominal task success.

Significance and Claims

The paper claims that compact RL policies can already reach VLA-like (Vision-Language-Action) multi-task breadth when trained with human demonstrations using the proposed infrastructure and algorithm.

Infrastructure: The work demonstrates that structured manipulation distributions can be compiled into heterogeneous GPU-parallel substrates, moving beyond isolated task runs to joint multi-task learning.
Algorithmic Efficiency: DGPO proves that demonstrations can be used effectively not just as fixed targets or offline data, but as adaptive, task-dependent guidance that preserves the stability of on-policy learning.
Future Outlook: The authors suggest that the same training interface (MT-Libero + DGPO) can extend to larger models, specifically for VLA+RL post-training, using the benchmark for large-scale interaction data and GPU-parallel infrastructure for efficient multi-task fine-tuning.

The paper remains modest regarding limitations, acknowledging that MT-Libero is currently restricted to simulated tabletop manipulation with fixed morphologies and that sim2sim transfer for contact-rich tasks remains imperfect. However, it establishes a foundational step toward scalable, generalist robot learning.