Technical Summary: RetrDex – Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand

Problem Statement

Retrieving objects buried beneath clutter is a fundamental challenge in embodied intelligence, particularly in unstructured environments like households. The core difficulty lies in the complex support relationships and occlusions that prevent robots from obtaining precise poses or feasible grasp points through passive vision alone. Existing approaches typically rely on a "perception-planning-action" paradigm that treats physical interactions (such as pushing) as isolated, sequential preparatory steps before grasping. These methods often fail to model scene exploration and target retrieval as a continuous, deeply coupled process. Furthermore, standard parallel-jaw grippers are limited to linear clearing motions, while even dexterous hand approaches often focus on discrete interaction-grasp combinations rather than learning a continuous spectrum of manipulation skills.

Methodology

The authors propose RetrDex, a framework designed to enable dexterous arm-hand systems to learn efficient object retrieval policies that transfer zero-shot from simulation to the real world. The methodology consists of four key components:

1. Task Construction and Environment

The system utilizes IsaacGym to generate diverse, realistic cluttered scenes. In each episode, 20+ household objects of varying masses, sizes, and shapes are dropped into a box, with a target object placed at the bottom. To ensure stable training, the robot hand alternates between a "prepare pose" (above the box) and a "suspending pose" (away from the box) to avoid interfering with object physics during initialization and to periodically compute pixel-based rewards without occlusion.

2. Spatially Aware Representation (Teacher Policy)

The core of the learning framework is a Teacher-Student architecture.

• Spatial Representation: A Graph Neural Network (GNN) module learns a spatially aware representation ($h^*_{t}$) that encodes occlusion patterns and spatial relationships among the target, the dexterous hand, and surrounding clutter. The graph includes the target, the hand, and the $K$ nearest neighboring objects. Nodes are encoded using PointNet++ on sampled point clouds, and EdgeConv layers propagate features to model context.
• Teacher Training: A teacher policy is trained using Proximal Policy Optimization (PPO) in parallel across 512 environments. It receives privileged information, including the GNN representation, full kinematic states of fingertips, and object velocities.
• Reward Design: The reward function encourages active exposure of the target through five components:
  • Distance Reward ($r_{dist}$): Encourages approaching occluded regions.
  • Stir Reward ($r_{stir}$): Promotes displacement of surrounding objects to clear deep occlusions.
  • Proximity Clearance Reward ($r_{clean}$): Focuses on clearing objects immediately adjacent to the target.
  • Pixel Visibility Reward ($r_{pixel}$): Rewards the number of visible target pixels in a top-down segmentation mask.
  • Penalties: Includes action, contact, and target displacement penalties.
  • Shaping: A potential-based reward shaping framework is used to accelerate training and reduce reward hacking.

3. Student Policy and Sim-to-Real Transfer

To enable deployment on real hardware, a Student Policy is distilled from the teacher via Behavior Cloning (BC).

• Input: The student observes only proprioceptive data (arm joint positions) and visual data (target pixel coordinates from a top-down camera), processed through a Transformer network that handles a history of 15 timesteps.
• Distillation: The teacher generates high-quality trajectories in simulation. Successful rollouts (where fingers avoid the box bottom and the target is evenly distributed) are filtered and used to train the student. This approach allows the student to learn complex temporal dependencies and manipulation skills (pushing, stirring, poking) without requiring the privileged spatial information available to the teacher.

Key Contributions

1. Unified Continuous Framework: RetrDex models object retrieval not as a sequence of discrete steps (push-then-grasp) but as a continuous, coupled process where the robot actively explores and manipulates clutter to expose the target.
2. Spatially Aware Graph Representation: The introduction of a GNN-based representation that explicitly encodes occlusion and spatial relationships enables the policy to reason about complex clutter structures and develop diverse manipulation strategies.
3. Zero-Shot Sim-to-Real Transfer: The framework successfully transfers policies trained in simulation to a real-world dexterous multi-fingered robot (Inspire Hand on a Realman RM-75 arm) without fine-tuning, demonstrating robustness to unseen objects and clutter configurations.
4. Skill Discovery: The system autonomously discovers effective skills such as stirring, poking, and reorienting, which are difficult to encode manually or learn with standard grippers.

Experimental Results

The authors evaluated RetrDex on 16 household objects across varied clutter configurations, comparing it against baselines including Visual-based Motion Planning (VMP), sequential Grasp-Pick, and variants using SAC or parallel-jaw grippers.

• Success Rate and Efficiency: RetrDex achieved a 91.4% retrieval success rate (RSR) on seen objects and 86.2% on unseen small objects in simulation, significantly outperforming baselines. It reduced retrieval steps by an average of 38% compared to VMP and 90% compared to Grasp-Pick.
• Generalization: The method demonstrated strong generalization to unseen object shapes (small and large) and increased clutter density (L2 and L3 settings), whereas performance for sequential removal methods dropped sharply as clutter complexity increased.
• Real-World Performance: In real-world experiments, RetrDex achieved high success rates across diverse object types (cuboids, cylinders, spheres) and positions. It reduced retrieval time by 51.2% compared to VMP and 61.9% compared to sequential grasp-and-remove methods.
• Ablation Studies:
  • Removing the GNN module or the "stir" reward significantly degraded performance, particularly on large objects and deep occlusions.
  • The dexterous hand variant vastly outperformed the parallel-jaw gripper variant, confirming the advantage of fine-scale manipulation skills (poking, stirring) over linear pushing.

Significance and Claims

The paper claims that RetrDex addresses a fundamental limitation in current robotic manipulation: the reliance on passive observation and sequential, discrete decision-making. By leveraging the dexterity of multi-fingered hands and large-scale reinforcement learning, the framework enables robots to autonomously discover efficient strategies for resolving occlusion.

The authors emphasize that the true advantage of dexterous hands lies in their capacity to learn a wide range of interactions that extend beyond discrete pre-grasp motions. The successful zero-shot transfer validates the practical applicability of learning complex, contact-rich manipulation skills in simulation for deployment in real-world, unstructured environments. The work suggests that treating occlusion resolution as a continuous, coupled process is a more effective paradigm than treating it as an auxiliary step.