Learning Reactive Dexterous Grasping via Hierarchical Task-Space RL Planning and Joint-Space QP Control

Technical Summary: Learning Reactive Dexterous Grasping via Hierarchical Task-Space RL Planning and Joint-Space QP Control

Problem Statement
Reactive dexterous grasping with multi-degree-of-freedom (DoF) hands remains a significant challenge due to hybrid dynamics, discontinuous contact modes, and complex frictional interactions. Traditional sequential pipelines (grasp synthesis, motion planning, trajectory tracking) struggle in dynamic, unstructured environments due to computational overhead and limited reactivity. While Reinforcement Learning (RL) offers a model-free alternative, monolithic end-to-end approaches face three critical limitations: (1) computationally expensive training that scales poorly with object diversity; (2) unpredictable behavior lacking formal safety guarantees; and (3) poor steerability, making it difficult to adapt to new obstacles or safety constraints without retraining. The authors argue these issues stem from placing an excessive learning burden on a single policy, forcing it to simultaneously solve high-level task reasoning, low-level kinematic coordination, and safety constraint satisfaction.

Methodology
The paper proposes a hybrid hierarchical control framework that explicitly decouples high-level spatial intent from low-level joint execution. The architecture consists of two distinct layers:

1. High-Level Multi-Agent RL Planner (Task-Space):

   • Inspired by human motor control, the planner is decomposed into two specialized agents trained within a Centralized Training with Decentralized Execution (CTDE) framework:
     • Arm Agent ($\pi_{arm}$): Responsible for global transport, outputting a desired palm twist (linear and angular velocities) in the world frame.
     • Hand Agent ($\pi_{hand}$): Responsible for local manipulation, outputting desired linear velocities for fingertips relative to the palm frame.
   • Observations: Agents utilize proprioceptive states and object states represented by a Minimum Volume Bounding Box (MVBB), avoiding the need for explicit 3D meshes or precomputed grasp poses.
   • Rewards: The system employs a shared grasp indicator and task-space lift rewards to encourage collaboration, alongside regularization for action smoothness.

2. Low-Level Joint-Space Constrained QP Controller:

   • The high-level task-space velocity commands are processed by a GPU-parallelized Quadratic Programming (QP) solver.
   • The QP maps task-space velocities to feasible joint velocities ($\dot{q}_{des}$) by minimizing tracking error subject to strict constraints:
     • Joint position and velocity limits.
     • Collision avoidance (robot-environment and self-collision).
     • Kinematic feasibility.
   • The QP output is converted to torque commands via a high-frequency PD controller.
   • Crucially, the QP solver is embedded directly into the RL training loop, ensuring the policy learns to operate within the physical constraints of the controller rather than learning to avoid them via reward shaping.

Key Contributions

1. Hybrid Hierarchical Architecture: A framework combining RL with an explicit physics-based controller, enabling data-efficient learning, explicit constraint handling, and steerable behavior.
2. Multi-Agent Decomposition: Demonstrating that separating the policy into palm-transport (arm) and finger-manipulation (hand) agents yields higher performance and faster convergence than monolithic policies, particularly for high-DoF hands.
3. GPU-Parallelized QP Solver: The development and open-sourcing of a GPU-parallelized QP solver integrated directly into the RL training pipeline, applicable to various arm-hand manipulation tasks.
4. Zero-Shot Transferability: Validation of robust grasping across diverse objects without explicit object models or precomputed poses, achieving successful sim-to-real transfer.

Experimental Results
The framework was validated through extensive simulations and real-world hardware experiments using a 7-DoF arm equipped with a 20-DoF anthropomorphic hand (ROBOTIS HX5-D20) and an 8-DoF 2-finger gripper.

• Simulation Performance: The Multi-Agent approach achieved an 81.4% success rate on the 5F hand, significantly outperforming End-to-End baselines (13.2%) and Single-Agent hierarchical baselines (80.9%). It also demonstrated superior pose tracking accuracy (4.2 cm position error, 15.9° orientation error) compared to other architectures.
• Sim-to-Real Transfer: The policy, trained purely in simulation, was deployed on physical hardware without fine-tuning. It successfully grasped and lifted 22 out of 26 unseen objects, including deformable items.
• Reactive Robustness: Hardware experiments demonstrated the system's ability to recover from unexpected physical disturbances (e.g., a spray bottle being pushed away during approach) and re-grasp displaced deformable objects (a toy sloth) by continuously recalculating task-space trajectories.
• Steerability: The framework supports zero-shot steerability. Operators can dynamically adjust safety margins (e.g., joint velocity limits) or add repulsive potential fields for obstacle avoidance without retraining the policy.

Significance and Claims
The authors claim that this structural isolation of task-space decision-making from low-level execution addresses the core limitations of pure RL approaches. By letting the physics-based QP handle tractable constraints (kinematics, safety), the RL policy is freed to focus exclusively on high-level strategic reasoning. This design:

• Accelerates training convergence by reducing the learning burden and avoiding conflicting reward signals.
• Ensures hardware safety through deterministic constraint enforcement rather than probabilistic reward shaping.
• Unlocks zero-shot steerability, allowing dynamic adjustment of safety margins and obstacle avoidance strategies post-training.
• Demonstrates robustness in unstructured environments, capable of reacting to physical disturbances and generalizing to unseen object geometries using only coarse spatial information (MVBB).

The paper concludes that this hybrid approach bridges the gap between the adaptability of learning-based methods and the safety guarantees of model-based control, providing a scalable solution for reactive dexterous manipulation.