Technical Summary: Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulation

Problem Statement

The development of generalist policies for bimanual dexterous manipulation is currently bottlenecked by the lack of large-scale, high-quality, and diverse datasets. While human teleoperation provides natural demonstrations, it is prohibitively expensive to scale across diverse tasks and objects. Human videos, though abundant, suffer from embodiment mismatches, partial observability, and retargeting imperfections that hinder robust policy learning. Conversely, simulation-based data generation offers scalability but often relies on object-centric trajectory matching derived from human demonstrations. This approach frequently neglects the physical feasibility of robot execution, leading to infeasible trajectories in complex, cluttered scenes. Furthermore, existing Reinforcement Learning (RL) approaches, while capable of generating diverse trajectories, are often constrained by the need for extensive, handcrafted, task-specific reward engineering, limiting their scalability.

Methodology: Reinforcement Learning as Data Collector (RLDC)

The authors propose RLDC, a systematic pipeline that leverages RL to generate high-quality synthetic training data, which is then distilled into a language-conditioned multi-task policy. The pipeline operates in four stages:

1. Generalizable Reward Design

To address the scalability issue of reward engineering, the authors design a universal set of four reward terms applicable to various bimanual tasks (e.g., box lifting, multi-object pick-and-place, drawer insertion):

• Initial Manipulation Hand Pose: Encourages alignment of the robot hand to a predefined pose relative to the object.
• Object Contact: Minimizes the distance between fingertips and the object's center of mass to ensure reliable contact initiation.
• Object Goal Poses with Lifting Gate: Rewards progress toward a flexible goal pose but only activates once the object is lifted above a specific height threshold. This prevents the agent from simply pushing objects rather than manipulating them in the air.
• Robot Reset: Encourages the robot to return to a safe, predefined configuration after task completion, facilitating task chaining.

These terms are combined with scaling factors to solve diverse tasks without requiring unique reward engineering for each.

2. Teacher Training and Data Collection

• Teacher Policies: Specialized RL teachers are trained for specific tasks using the generalizable rewards. The bimanual problem is formulated as a multi-agent setting using Independent PPO (IPPO), where each arm is controlled by a dedicated policy conditioned on local observations.
• Domain Randomization: Once teachers converge, they are rolled out in simulation with extensive randomization. This includes variations in object geometry (100+ complex objects), physical parameters (friction, mass, restitution), visual attributes (color, reflectivity), and perception factors (camera viewpoints, lighting).
• Automated Language Labeling: Language annotations are automatically generated using templates based on object properties and goals. These templates are further augmented using a Vision-Language Model (VLM) to paraphrase instructions, enriching the diversity of natural language labels (e.g., "Pick the nearest objects" vs. "Use the right arm to pick A...").

3. Multi-Task Policy Distillation

The collected synthetic dataset is distilled into a single, unified multi-task policy using a diffusion model architecture:

• Input: The model processes a 6-channel point cloud (RGB-D), robot state (joint positions/velocities), and language tokens.
• Architecture: A transformer-based network encodes point clouds using PointNet++ (without max pooling to preserve spatial structure) and robot states using Fourier positional encodings.
• Decoupled Action Heads: To handle the partially independent yet coordinated nature of bimanual tasks, the action representation is decoupled into separate queries for the left and right arms. These interact with conditioning tokens via cross-attention.
• Auxiliary Loss: An auxiliary head predicts ground-truth object states (positions and orientations) from the point cloud features. This anchors the visual encoder to underlying state information, a capability not available with human video data alone.

4. Evaluation Setup

The system is evaluated on three tasks: box lifting, multi-object pick-and-place, and drawer insertion. Experiments are conducted in NVIDIA IsaacLab simulation and on a real-world platform consisting of two xArm UF850 arms equipped with 16-DOF Allegro Hands.

Key Results

• Reward Design Efficiency: The proposed generalizable reward design achieves success rates comparable to finely tuned task-specific rewards and outperforms object-reference trajectory baselines, particularly in the "Lift" task. It significantly reduces engineering overhead, requiring approximately one hour per task compared to a full day for trajectory-based rewards.
• Policy Performance: The distilled multi-task policy significantly outperforms state-of-the-art baselines (DP3 and iDP3) in both simulation and real-world settings.
  • Simulation: Achieved success rates of 0.80 (Multi-Object), 0.74 (Insert), and 0.96 (Lift), compared to baselines ranging from 0.06 to 0.70.
  • Real-World: Achieved success rates of 15/20 (Multi-Object), 16/20 (Insert), and 14/20 (Lift).
• Zero-Shot Transfer: The policy successfully transfers to real-world settings and demonstrates zero-shot generalization to out-of-distribution objects (e.g., inserting a "dog" object not seen in the specific insertion training set but seen in the multi-object training set).
• Ablation Studies:
  • Separate Heads: While a shared head performed slightly better in simulation, separate action heads significantly improved robustness in the real world, particularly for handling out-of-distribution cases where one arm's failure does not compromise the other.
  • Auxiliary Loss: Including the state-prediction auxiliary loss consistently improved performance across all tasks, validating the benefit of anchoring visual representations to state information.
  • Dataset Size: Performance scales effectively with dataset size, with larger datasets better capturing rare corner-case initializations.

Significance and Claims

The paper claims that RLDC provides a scalable, automated pipeline for generating large-scale, multimodal datasets for bimanual dexterous manipulation. Its primary significance lies in:

1. Scalability: Moving away from expensive human teleoperation and rigid trajectory matching toward an RL-driven approach that can autonomously generate feasible, diverse trajectories.
2. Generalizability: Demonstrating that a single set of reward terms can effectively train policies for diverse, complex bimanual tasks, reducing the engineering burden of reward design.
3. Sim-to-Real Transfer: Proving that policies trained entirely on synthetic data generated by this pipeline can achieve strong performance on real-world dexterous hands without fine-tuning.
4. Multimodal Integration: Successfully integrating language conditioning and visual grounding (via point clouds and auxiliary state prediction) to create robust, instruction-following robotic policies.

The authors acknowledge limitations, noting that setting up simulation environments and initial poses remains time-consuming and that RL-generated trajectories may lack the smoothness or "human-like" quality required for delicate manipulation tasks, suggesting future work could combine RL data with human demonstrations.