Object Pose and Shape Estimation for Grasping: Does it Work?

Technical Summary: Object Pose and Shape Estimation for Grasping

Problem Statement
The paper addresses a fundamental question in robotics: Have recent advancements in object pose and shape estimation matured sufficiently to outperform state-of-the-art end-to-end grasp synthesis methods? Traditionally, grasp synthesis relied on known object geometry or analytical optimization. While modern end-to-end methods (e.g., AnyGrasp) directly regress grasp poses from single-view RGB-D images with high speed, they often struggle with stability and force closure. Conversely, modular approaches—where object geometry is first reconstructed and then used for antipodal sampling—have historically been limited by the ambiguity of single-view reconstruction. This work investigates whether modern encoder-decoder and diffusion-based reconstruction models can now serve as a superior foundation for grasp synthesis compared to direct regression.

Methodology
The authors scope their study to parallel-jaw grippers, 7-DoF grasps, and single-view RGB-D input. They implement and compare four baselines:

1. End-to-End Baseline: AnyGrasp, a state-of-the-art model that directly regresses 7-DoF grasp poses from partial point clouds without explicit geometry reconstruction.
2. Modular Baselines: Three methods that first estimate the object pose and shape for all objects in the scene, render a 3D mesh (via marching cubes), and then synthesize grasps using antipodal sampling (ensuring force closure) and collision filtering. These modular methods utilize two distinct reconstruction paradigms:
   • Encoder-Decoder Models: CRISP-Grasp (using CRISP) and SAM3D-Grasp (using SAM3D). These models encode RGB-D images into shape embeddings or normalized coordinates to decode object meshes and poses.
   • Diffusion-Based Models: SC-Grasp (using SceneComplete). This pipeline leverages InstantMesh (a diffusion model for novel-view synthesis) augmented with BrushNet for occlusion handling and FoundationPose for pose estimation to reconstruct multi-object scenes.

The evaluation is conducted across three environments: a physics simulator (PyBullet with YCBV objects), real-world datasets (YCBV and NOCS test sets), and physical experiments using an Xarm7 manipulator with a parallel-jaw gripper. Metrics include Grasp Collision Rate (GCR), Force Closure Failure Rate (FCFR), and Grasp Stability (GS).

Key Contributions

• Comparative Analysis: The paper provides a rigorous experimental comparison between direct end-to-end grasp synthesis and a modular pipeline combining modern reconstruction with antipodal sampling.
• Performance Validation: It demonstrates that modular methods consistently outperform end-to-end methods in grasp success rates, stability, and force closure satisfaction across simulated and real-world settings.
• Failure Mode Analysis: The authors dissect the failure modes of modular methods, identifying that incorrect scale estimation is the primary cause of failure (38–50% of cases), followed by pose and shape errors in diffusion-based models.
• Language-Conditioned Grasping: The work extends the modular pipeline to language-conditioned grasping. By reconstructing a "digital twin" from a single view and feeding it into the LERF-TOGO pipeline, the authors achieve performance comparable to multi-view baselines, enabling semantic grasp selection from single-view inputs.

Results

• Success Rates: In physics simulations, modular methods achieved a grasp success rate 1.6 to 2 times higher than AnyGrasp. In real-world datasets, AnyGrasp achieved only 10–11% success, whereas modular methods performed significantly better.
• Stability and Force Closure: The end-to-end method exhibited a high Force Closure Failure Rate (61.3% in simulation) and produced unstable grasps in real-world trials (e.g., post-lift rotations). Modular methods, by explicitly enforcing force closure via antipodal sampling on reconstructed meshes, yielded stable grasps in 100% of trials for several objects in physical experiments.
• Clutter and Occlusion: Performance for all methods degraded as scene clutter increased. However, modular methods (specifically CRISP-Grasp) maintained a performance advantage over end-to-end methods even in scenes with 10 objects. The diffusion-based SC-Grasp showed a larger performance drop in highly occluded scenes due to limitations in its in-painting model.
• Run-Time: The end-to-end method (AnyGrasp) was the fastest (0.33s). Modular methods were significantly slower (45–105s total), with the bottleneck being the pose/shape estimation (especially for diffusion models) and the antipodal sampling process.
• Filtering Efficacy: Applying pose and shape estimates to filter grasps generated by the end-to-end method (AnyGrasp+CRISP/SC) drastically reduced the number of candidate grasps but significantly increased the success rate, confirming that reconstruction quality is a strong indicator of grasp feasibility.

Significance and Claims
The paper claims that the field has reached an "inflection point." While previous literature (e.g., Sundermeyer et al., 2021) argued that learned single-view reconstructions were too ambiguous and coarse for reliable grasp planning, this work demonstrates that with current state-of-the-art reconstruction models (SAM3D, CRISP, SceneComplete), the modular approach is now superior to end-to-end regression for grasp synthesis in terms of robustness and stability.

The authors modestly conclude that while modular methods outperform end-to-end approaches in quality, their adoption is currently limited by computational run-time requirements. They assert that the accuracy of the downstream grasp task is contingent on the accuracy of the upstream pose and shape estimation, particularly regarding scale and occlusion handling. Furthermore, the work highlights the viability of using single-view reconstruction as a bridge for vision-language models, enabling semantically meaningful, language-conditioned grasping without the need for multi-view scanning.