Technical Summary: GAPartManip

Problem Statement

Effective manipulation of articulated objects (e.g., microwaves, cabinets, washing machines) in household scenarios is a critical step toward general embodied artificial intelligence. However, current 3D vision and robotics research faces two primary limitations when addressing these objects:

1. Material-Agnostic Depth Perception Challenges: Mainstream methods relying on point clouds and depth perception often fail in real-world environments due to imperfect depth estimation on challenging materials. Transparent lids, reflective handles, and specular surfaces cause significant gaps between simulation and reality (sim-to-real), leading to missing geometric structures like handles and knobs. Existing neural stereo-matching methods, while successful on rigid objects, lack the diversity in stereo infrared (IR) datasets required to generalize to large, complex articulated objects.
2. Lack of Actionable Part-Based Interaction Poses: Existing approaches struggle to predict stable, actionable interaction poses across different object categories.
   • Heuristic-based methods lack the geometric diversity and detail required for robust real-world interactions.
   • Rigid object grasp predictors fail to distinguish between independent links in articulated objects, often generating non-actionable poses.
   • Affordance-based methods generate heatmaps that are often ambiguous, difficult to annotate, and fail to produce stable, specific interaction poses.

Methodology

The paper introduces GAPartManip, a large-scale, part-centric synthetic dataset designed to address these limitations from a data-centric perspective. The methodology encompasses dataset generation, a novel manipulation framework, and extensive evaluation.

1. The GAPartManip Dataset

The dataset is constructed to support material-agnostic articulated object manipulation. It inherits 918 object instances across 19 common household categories (e.g., Box, Door, Microwave, Refrigerator) from the previous GAPartNet dataset.

• Photo-Realistic Rendering: Built on NVIDIA Isaac Sim, the pipeline simulates the Intel RealSense D415 structured light camera system (RGB, Left IR, Right IR). It employs domain randomization to mimic real-world conditions, varying ambient lighting, backgrounds, and crucially, the physical material properties (diffuse, transparent, specular, metal) of object parts.
• Scale: The dataset contains 241,680 rendering samples and approximately 8 billion scene-level, part-oriented actionable interaction pose annotations.
• Pose Annotation Pipeline:
  • Part-Level: Meshes for each part are fused, and Farthest Point Sampling (FPS) is used to generate candidate points. Dense stable interaction poses are sampled based on antipodal analysis.
  • Scene-Level: Part-level poses are projected into the scene. A GPU-accelerated CUDA-based filtering process removes unreasonable (misaligned with partial point clouds) and unreachable (collision-inducing) poses. This optimization reduced annotation time from a projected year to 3 days.

2. Proposed Framework

The authors propose a modular framework for articulated object manipulation consisting of three components:

• Depth Reconstruction Module: Addresses the issue of incomplete or incorrect raw sensor depth caused by challenging materials. It utilizes a diffusion model-based approach (specifically fine-tuning D3RoMa) to estimate and restore depth maps using raw sensor depth and IR images.
• Pose Prediction Module: Adapted from EconomicGrasp, this module is renamed Part-aware EcoGrasp. Unlike standard grasp predictors, it estimates "actionability" rather than just "graspness." It predicts:
  • A 7-DoF actionable interaction pose (where to grasp).
  • A 6-DoF interaction motion (post-grasping movement), predicted using a pre-trained GAPartNet.
  • The model is trained to identify points on actionable parts and filter poses that cause collisions.
• Local Planner Module: Utilizes CuRobo to optimize motion trajectories to the predicted actionable poses, compute inverse kinematics, and execute the grasping and subsequent interaction motions.

Key Contributions

1. GAPartManip Dataset: The introduction of a novel, large-scale synthetic dataset featuring realistic, physics-based IR rendering for diverse materials and extensive, part-oriented, scene-level actionable interaction pose annotations (8 billion poses).
2. Novel Manipulation Framework: A modular system integrating depth reconstruction and part-aware pose prediction, demonstrating superior effectiveness and robustness compared to baseline methods.
3. Empirical Validation: Comprehensive experiments in both simulation and the real world, achieving State-of-the-Art (SOTA) performance in depth estimation, actionable pose prediction, and overall articulated object manipulation tasks.

Experimental Results

Depth Estimation

• Simulation: Fine-tuning the D3RoMa model on GAPartManip resulted in a 600% improvement in Mean Absolute Error (MAE) compared to its pre-trained version, significantly outperforming traditional stereo matching (SGM) and other learning-based baselines (RAFT-Stereo).
• Real-World: The fine-tuned model demonstrated robust depth recovery for transparent and translucent lids and small handles, outperforming raw depth sensors and other fine-tuned baselines.

Actionable Pose Prediction

• Simulation: The proposed Part-aware EcoGrasp, trained on GAPartManip, achieved a precision of 55.33% on seen instances and 41.65% on novel instances. This substantially outperformed fine-tuned GSNet (25.70% seen) and Where2Act (14.43% seen).
• Analysis: The dataset provided strong geometric priors, enabling the network to focus on actionable parts rather than non-actionable ones, a distinction pre-trained models often failed to make.

Real-World Manipulation

• Performance: In real-world experiments involving 7 distinct instances (StorageFurniture, Box, Microwave) and 31 tasks (open/close), the proposed framework achieved an overall success rate of 61.29%.
• Comparison: This significantly outperformed baselines: AO-Grasp (29.03%), Where2Act (19.35%), and GSNet (32.25%).
• Ablation: Removing the depth reconstruction module dropped the success rate to 38.70%, and removing the part-aware pose estimator dropped it to 51.61%, confirming the necessity of both modules.

Significance and Claims

The paper claims that GAPartManip is the first dataset to feature photo-realistic material randomization combined with detailed, part-oriented, scene-level actionable interaction pose annotations for articulated objects.

The authors assert that their work addresses the critical sim-to-real gap in articulated object manipulation, particularly regarding imperfect point clouds caused by challenging materials. By providing a data-centric solution, they demonstrate that robust zero-shot transfer to the real world is achievable. The proposed framework not only improves individual modules (depth and pose prediction) but also delivers a cohesive system that achieves SOTA performance in generalizable articulated object manipulation, paving the way for more flexible and adaptable embodied AI in household scenarios.