MoRI: Mixture of RL and IL Experts for Long-Horizon Manipulation Tasks

1. Problem Statement

Robotic manipulation, particularly for long-horizon tasks involving both coarse movements and contact-rich precision, faces a fundamental trade-off between Imitation Learning (IL) and Reinforcement Learning (RL):

• IL (e.g., Behavioral Cloning): Offers sample efficiency and fast policy derivation from demonstrations but suffers from compounding errors and distribution shift, making it fragile in high-precision or out-of-distribution scenarios.
• RL: Facilitates autonomous exploration and robustness in contact-rich tasks but is hindered by low sample efficiency, high computational costs, and the need for extensive human intervention to ensure safety and convergence.
• Existing Hybrids: Current methods that combine IL and RL often struggle with complex, long-horizon tasks, failing to balance efficiency, safety, and convergence speed effectively.

The core challenge is to create a framework that leverages the efficiency of IL for stable motions while utilizing the adaptability of RL for fine-grained corrections, all while minimizing human intervention and training time.

2. Methodology: The MoRI Framework

The authors propose MoRI (Mixture of RL and IL Experts), a system based on the Mixture-of-Experts (MoE) architecture. It dynamically switches between an IL expert and an RL expert based on the variance of expert actions.

A. Architecture Overview

The system consists of three primary policy components:

1. BC Expert (IL): A Behavioral Cloning policy trained on demonstrations to handle coarse, deterministic motions (e.g., opening a drawer).
2. RL Expert: A Soft Actor-Critic (SAC) policy designed for fine-grained, contact-rich manipulations (e.g., inserting a plug) where exploration is necessary.
3. DBC Expert (Discrete BC): A specialized policy for discrete gripper control (open/hold/close) to accelerate learning in discrete action spaces.
4. Gating Network: A neural network that evaluates the state $s$ and outputs weights ($w_{bc}, w_{rl}$) based on the variance of the actions predicted by the BC and RL experts.

B. Two-Stage Training Pipeline

The framework operates in two distinct stages to ensure safety and efficiency:

Stage 1: Advantage-Weighted Offline Pre-training

• Goal: Initialize policies using a small set of demonstration data (20 samples) without online interaction.
• BC Policy: Fine-tuned via Mean Squared Error (MSE) loss on demonstrations.
• RL Policy: Initialized using Advantage-Weighted Actor-Critic (AWAC). This balances demonstration knowledge with exploratory capabilities, mitigating the conservatism of offline RL.
• DBC Policy: Trained via cross-entropy loss on discrete gripper actions to avoid the slow convergence of standard DQN in long-horizon tasks.
• MoE Gating: Remains untrained during this stage to prevent overfitting to limited data.

Stage 2: Mixture-of-Experts Driven Online Fine-tuning

• Goal: Refine policies in the real world with minimal human intervention.
• Data Buffers: Utilizes three buffers: Demo Buffer (initial + interventions), Success Buffer (successful online trajectories), and Replay Buffer (real-time interactions).
• Dynamic Scheduling: The gating network selects the final action based on expert variance:
  • If $w_{bc} > w_{rl}$, the system selects the BC action (low variance implies high confidence in the imitation).
  • Otherwise, it selects the RL action (high variance implies the need for exploration/correction).
• Loss Functions:
  • MoE Loss: Combines weighted variance minimization, load balancing, expert specialization, and entropy regularization to optimize the gating mechanism.
  • RL Regularization: The RL actor loss includes a term $\beta \|\pi_{rl}(s) - \pi_{bc}(s)\|^2$. This IL-based regularization constrains the RL exploration to remain within the distributional support of the BC policy, ensuring safety and preventing catastrophic divergence.

3. Key Contributions

1. Dynamic Scheduling Mechanism: A novel MoE framework that coordinates IL and RL experts based on action variance. This allows the system to automatically assign coarse movements to IL and fine-grained manipulations to RL, compensating for the weaknesses of each.
2. Hybrid Training Pipeline: Integrates offline pre-training (using AWAC and BC) with online fine-tuning. This approach accelerates convergence and reduces the reliance on large-scale human demonstrations.
3. Safety via Regularization: Introduces IL-based regularization for the RL component. This maintains exploration safety, improves sample efficiency, and significantly reduces the frequency of human intervention during training.
4. Discrete Gripper Control: Replaces standard DQN with a Discrete BC (DBC) loss for gripper actions, solving convergence issues in long-horizon tasks involving frequent discrete state changes.

4. Experimental Results

The method was evaluated on four complex real-world tasks using a Franka Research 3 robotic arm:

1. Place Block in Drawer
2. Put Towel in Lidded Box
3. Insert Two Sockets
4. Double-Fold the Towel

Key Performance Metrics:

• Success Rate: MoRI achieved an average success rate of 97.5% after 2–5 hours of fine-tuning.
  • Comparison: Significantly outperformed the baseline ConRFT (57.5%) and individual sub-experts (BC: 37.5%, RL: 77.5%).
• Training Efficiency: Reduced average training time by 21% compared to ConRFT.
• Human Intervention: Reduced human intervention by 85.8% (average intervention ratio dropped from 49.6% in ConRFT to 7.0% in MoRI).
• Autonomous Success: Increased the proportion of autonomous successful samples by 111.7%.
• Episode Length: MoRI achieved shorter episode lengths (avg. 161.4 steps) compared to BC (206.0) and RL (174.9), indicating more efficient task execution.

Ablation Studies:

• BC Regularization: Removing the IL regularization caused the success rate to drop to 70% and human intervention to skyrocket to 45.4%, confirming its critical role in safety and convergence.
• Gripper Control: Using DBC instead of DQN improved success rates from 72.5% to 97.5% and reduced training time significantly.

5. Significance

MoRI represents a significant advancement in robotic manipulation by effectively bridging the gap between the stability of Imitation Learning and the adaptability of Reinforcement Learning.

• Practical Deployment: By achieving high success rates with minimal human intervention and short training times (2–5 hours), MoRI addresses the "sample inefficiency" bottleneck that has historically hindered the deployment of RL in real-world robotics.
• Safety: The use of action variance for gating and IL-based regularization provides a robust mechanism for safe exploration, making it suitable for contact-rich and high-precision tasks where hardware damage is a risk.
• Generalizability: The framework demonstrates that a hybrid MoE approach can dynamically adapt to varying task requirements (coarse vs. fine), offering a scalable solution for complex, long-horizon robotic tasks.