HOGraspFlow: Taxonomy-Aware Hand-Object Retargeting for Multi-Modal SE(3) Grasp Generation

Technical Summary of HOGraspFlow

Problem Statement
The paper addresses the challenge of retargeting human hand-object interaction (HOI) demonstrations to robotic parallel-jaw (PJ) end-effectors. Existing approaches often rely on a simplified "thumb-index" proxy, which collapses the diversity of human grasp taxonomy and neglects contact-dependent force closure, limiting them to pinch-like grasps. Conversely, recent multi-embodiment grasp generation methods are typically object-centric, requiring reliable 3D geometric priors (e.g., SDFs, point clouds) and pose estimation at test time. These geometric dependencies make them brittle to sensing artifacts (e.g., missing depth, specular reflections) and unable to directly exploit "in-the-wild" video data where object geometry is unknown or unreliable. The core problem is to develop a vision-based framework that can generate multi-modal, executable 6-DoF PJ grasps from a single RGB frame without explicit object models, while preserving the diversity of human grasp intents and contact semantics.

Methodology
The authors propose HOGraspFlow, an affordance-centric retargeting framework that synthesizes SE(3) grasp poses using denoising flow matching (FM). The pipeline operates as follows:

1. Perception and Feature Extraction:

   • The system takes a single RGB image of an HOI demonstration.
   • It utilizes WiLoR, a foundational hand reconstructor, to extract MANO parameters (pose and shape) and hand bounding boxes.
   • Instead of relying on object meshes, the method extracts visual semantics from the hand-object crop using DINOv2, a vision foundation model, to obtain patch tokens that are invariant to object appearance and category.
   • These visual features are fused with hand parameters via self-attention to create a compact HOI-aware descriptor.

2. Conditioning Cues:
   The generative process is conditioned on three complementary cues derived from the descriptor:

   • Visual Semantics: The DINOv2 features provide context about the interaction.
   • Contact Reconstruction: A lightweight decoder predicts per-vertex contact logits on the hand mesh, answering "where to grasp" without requiring explicit contact input.
   • Taxonomy-Aware Prior: A grasp type classifier (trained on the GRASP taxonomy) outputs a probability distribution over 33 grasp types. This is mapped to a learnable codebook to provide a categorical prior, answering "how to grasp" and regularizing the distribution of retargeted poses.

3. Generative Synthesis (SE(3)):

   • The framework introduces two generative models: HOGraspDiff (based on score-based diffusion) and HOGraspFlow (based on flow matching).
   • Both models operate on the SE(3) manifold to generate diverse, multi-modal PJ grasp poses.
   • HOGraspFlow is highlighted for its stability. Unlike score-based models that learn a score function under a variance-exploding perturbation (leading to scale drift), Flow Matching learns a deterministic velocity field. This decouples translation and rotation, reducing optimization instability and improving sample efficiency.
   • Guidance: The generation process incorporates soft guidance to align the gripper's approach axis with the hand's palm axis and uses classifier-free guidance to steer the distribution.

4. Deployment:

   • Generated poses are initially in the hand's wrist frame.
   • To map to the world frame, the system uses the global hand pose from WiLoR.
   • A "Z-only ICP" (Iterative Closest Point) refinement is applied to correct translation bias along the camera ray, compensating for monocular depth estimation errors without requiring multi-view setups.

Key Contributions

1. Vision-Based Affordance-Centric Framework: A novel approach that generates multi-modal 6-DoF PJ grasps from a single RGB frame without requiring explicit 3D object geometry or pose estimation, relying instead on foundational visual features and hand reconstruction.
2. Dual Generative Architectures: The introduction of HOGraspDiff and HOGraspFlow for vision-based SE(3) retargeting. The paper demonstrates that Flow Matching offers superior distributional fidelity and optimization stability compared to Score-based Diffusion in this context.
3. Taxonomy and Contact Integration: The method explicitly integrates grasp taxonomy recognition and contact reconstruction as conditioning signals, allowing the system to recognize diverse grasp types and maintain high contact accuracy without ground-truth contact inputs during inference.
4. Real-World Validation: Successful deployment on a UR10e robot with a Robotiq 2F-85 gripper, demonstrating robust transfer to diverse objects (including out-of-distribution items) with high success rates.

Results

• Simulation (HOGraspNet Dataset):
  • HOGraspFlow outperformed HOGraspDiff and other baselines. With DINOv2 and the taxonomy codebook, HOGraspFlow achieved a Taxonomy Accuracy (TA) of 92.5% and Contact Accuracy (CA) of 88.1%.
  • The Earth Mover's Distance (EMD) for HOGraspFlow was 0.67, significantly lower (better) than the score-based variant (0.80) and the baseline without visual features (0.81).
  • Ablation studies confirmed that visual semantics (DINOv2) are critical for semantic understanding, outperforming a "contact-oracle" (which uses ground-truth contact labels) in grasp type classification, suggesting visual cues are more discriminative for taxonomy than contact alone.
• Real-World Experiments:
  • The system achieved an overall grasp success rate of 83.8% (67/80 trials) across various objects, including tools and household items.
  • It significantly outperformed a standard thumb-index template approach (33/80) and a point-cloud-based GraspNet-1B baseline (53/80), particularly in handling objects with challenging geometries or sensing artifacts.
  • The method demonstrated robustness to occlusions and specularities where geometry-centric methods failed.

Significance
The paper claims that HOGraspFlow represents a shift from object-centric to intent-centric grasp generation. By leveraging the asymmetry between high object variability and constrained human grasp intents, the method achieves reliable retargeting without the brittleness of geometric priors. The authors emphasize that Flow Matching provides a more stable and efficient optimization path for SE(3) pose generation compared to traditional diffusion models. The work demonstrates that high-fidelity, multi-modal grasp synthesis is achievable directly from monocular video, enabling robots to learn from diverse, in-the-wild human demonstrations without the need for precise 3D reconstruction of the target objects.