GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors

Technical Summary: GraspFoM

Problem Statement
Robotic grasping remains a significant challenge in real-world scenarios characterized by partial observations, occlusion, and clutter. Effective grasping requires reasoning about both local contact cues and the global 3D structure of an object. While existing geometry-aware methods recognize the value of 3D reconstruction, they typically treat geometry as an intermediate prediction rather than a reusable object prior. Furthermore, current unified frameworks often rely on task-specific representations learned from scratch or depend on Vision-Language Models (VLMs) that inherit 2D semantic knowledge, leading to slow inference and poor 3D pose alignment. There is a gap between general 3D reconstruction models, which optimize for asset fidelity, and the specific manipulation-oriented structures (e.g., graspable regions, affordances) required for robotic grasping.

Methodology: GraspFoM
The paper proposes GraspFoM, a unified framework that leverages 3D foundation priors (specifically SAM3D) to construct a shared 3D object latent for both object reconstruction and grasp pose prediction. The pipeline operates as follows:

1. Shared Object Latent Preparation:

   • The system takes RGB images, object masks, and point maps as input.
   • It utilizes the sparse-structure stage of SAM3D to initialize a shared 3D object latent ($z$) and coordinate alignment parameters.
   • This latent is converted into a structured 3D feature volume and lifted into a shape-aware grid.
   • For each queried surface point, the model fuses local geometric cues (point position and normal) with global shape context extracted from the latent grid via cross-attention. This creates a fused point token that encodes both local surface details and global object structure.

2. Anchor-Initialized Pose-Reasoning Diffuser:

   • Instead of regressing a single pose or relying on discrete grasp candidates, GraspFoM models grasp prediction as a latent-conditioned denoising process in a normalized pose space.
   • A set of grasp anchors is created by clustering valid training poses.
   • The model employs a truncated anchored diffusion process. Rather than starting from pure Gaussian noise, the diffusion process is initialized near the nearest anchor, improving denoising efficiency while maintaining the ability to model multimodal grasp distributions.
   • The diffuser predicts continuous 6D rotation and grasp depth directly from the shared object latent.

3. Reconstruction-Aware Scoring and Latent Refinement:

   • Scorer: A reconstruction-aware scorer predicts three point-wise metrics: graspness (feasibility), grasp quality, and affordance. These scores are grounded in the reconstructed object structure rather than pose-only cues.
   • Residual Latent Updater: To enforce bidirectional interaction, grasp supervision is fed back into the reconstruction branch. A confidence-weighted summary of grasp signals is used to predict a residual feature volume, which is added to the original latent grid.
   • Regularization: A surface-aware regularization loss ensures the refined latent maintains geometric consistency while aligning with manipulation-relevant cues.

4. Inference:

   • The system performs truncated DDIM sampling from grasp anchors to generate continuous pose hypotheses.
   • Grasp hypotheses are ranked using a composite score combining graspability, view-alignment, and anchor probability.
   • The same shared latent is decoded to output high-fidelity 3D assets in mesh or 3D Gaussian (3DGS) formats alongside the grasp poses.

Key Contributions

• Unified Framework: The proposal of GraspFoM, which leverages 3D foundation priors to build a shared 3D object latent for simultaneous object reconstruction and grasp pose prediction.
• Continuous Pose Generation: The introduction of an anchor-initialized, pose-reasoning diffuser that models continuous, multimodal grasp pose distributions, avoiding reliance on discrete grasp candidates.
• Bidirectional Interaction: The design of a reconstruction-aware scorer and a residual latent updater that couples reconstruction and grasp learning, enabling geometric grounding and the enhancement of grasp-relevant affordances within the shared latent.
• Dual Output: The capability to jointly predict accurate grasp poses and reconstruct high-fidelity 3D assets (mesh and 3DGS) with state-of-the-art performance.

Experimental Results
The authors evaluated GraspFoM on the GraspNet-1Billion benchmark:

• Grasp Pose Prediction: GraspFoM achieved state-of-the-art results across all splits (Seen, Similar, Novel) and friction settings. Notably, it showed significant improvements on the Novel split (unseen object categories), suggesting strong generalization via reusable object-level priors. It outperformed existing unified methods like ZeroGrasp and MG-Grasp, despite being trained only on the official GraspNet-1B split without additional synthetic datasets.
• 3D Reconstruction: The framework achieved superior geometry quality compared to sparse-structure baselines (MinkowskiNet, OCNN, OctMAE) and the unified ZeroGrasp, achieving the best Chamfer Distance (CD), F1-Score, and Normal Consistency (NC).
• Ablation Studies: Experiments confirmed the necessity of the anchor number ($K=8$), diffusion horizon ($T_{max}=50$), and refinement steps ($N_{ref}=4$). Removing the reconstruction update module significantly degraded both reconstruction quality and grasp performance, validating the mutual reinforcement between the two tasks.
• In-the-Wild Evaluation: Qualitative results on the WildRGB-D dataset demonstrated more complete reconstructions and stable grasp poses compared to ZeroGrasp.

Significance and Claims
The paper claims that GraspFoM addresses the limitation of existing methods that treat geometry as a transient intermediate step. By leveraging transferable 3D foundation priors (SAM3D), the method establishes a shared representation that is both geometrically faithful and manipulation-aware. The authors argue that this approach provides a more data-efficient route to robust grasp generalization compared to learning task-specific representations from scratch. The bidirectional link between reconstruction and grasping allows the system to refine object representations based on manipulation needs while using grounded geometric cues to improve grasp prediction. The framework achieves state-of-the-art performance on both tasks with only a small number of additional trainable parameters.

Limitations
The authors note that the current framework primarily focuses on rigid objects. They identify the incorporation of structured kinematic priors (e.g., URDF) for articulated object manipulation as a direction for future work.