GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting

1. Problem Statement

Reconstructing realistic 3D hand-object interactions from a single monocular RGB video is a fundamental challenge for Augmented/Virtual Reality (AR/VR), robotics, and embodied AI. Existing methods face three primary limitations:

• Category Specificity: Template-based approaches require pre-defined object shapes, failing to generalize to novel objects.
• Computational Cost: Category-agnostic methods (e.g., NeRF-based pipelines like HOLD) often require hours of optimization per sequence, limiting practical applicability.
• Physical Inconsistency: Current methods struggle with severe occlusions (hands hiding objects), leading to physically implausible contacts, missing geometry, and unstable scales.

2. Methodology: GHOST Framework

GHOST (Gaussian Hand-Object SplaTting) is a three-stage pipeline designed to reconstruct dynamic, animatable hand-object interactions efficiently using 2D Gaussian Splatting.

Stage 1: Preprocessing & Geometric Prior Retrieval

• Segmentation & SfM: The pipeline segments objects and hands using SAM2 and estimates camera poses and sparse point clouds using Structure-from-Motion (SfM). The authors evaluate both HLoc/COLMAP and the video-based VGGSfM, finding VGGSfM superior for certain datasets.
• Geometric Prior Retrieval: To handle occluded regions, GHOST retrieves a 3D geometric prior from the Objaverse database.
  • A Vision-Language Model (e.g., InternVL) generates a text description of the object.
  • OpenShape's CLIP model maps this text to a shared embedding space to retrieve $k$ candidate meshes.
  • Prior-Mask Alignment: An affine transformation (rotation $R_p$, translation $T_p$, scale $S_p$) is optimized to align the retrieved mesh silhouette with the object's segmentation mask, minimizing Intersection-over-Union (IoU) loss.

Stage 2: Hand-Object (HO) Alignment

Initial hand reconstructions (via HaMeR) and object point clouds often suffer from scale drift and misalignment. GHOST refines these using grasp-aware reasoning:

• Grasp Detection: The system analyzes motion trajectories to identify frames where hands are interacting with the object (based on cosine similarity of motion vectors).
• Optimization Objective: During grasping frames, a joint optimization minimizes three losses:
  1. Contact Loss ($L_{contact}$): Enforces the hand mesh to remain close to the object surface (Chamfer Distance).
  2. 2D Projection Loss ($L_{proj}$): Ensures 3D hand joints reproject correctly to 2D observations.
  3. Temporal Consistency Loss ($L_{temp}$): Smooths hand translations over time.

Stage 3: Gaussian Splatting Optimization

The core reconstruction uses 2D Gaussian Splatting to represent both hands and objects as dense, view-consistent Gaussian discs.

• Object Optimization:
  • Hand-Aware Background Loss ($L_{bkg,h}$): Unlike standard background losses that penalize Gaussians outside the object mask, this loss combines hand and object masks. This prevents the system from incorrectly removing object regions occluded by fingers.
  • Geometric Consistency Loss ($L_{geo}$): Uses the retrieved geometric prior to guide reconstruction. It penalizes Gaussians drifting too far from the prior surface ($L_{out}$) and penalizes large gaps on the prior surface ($L_{fill}$), effectively "filling in" occluded holes.
• Hand Optimization:
  • Hands are represented as deformable Gaussians attached to a canonical MANO mesh.
  • Rigging: Gaussians are rigged to mesh faces using local affine transformations ($T_{aff}$) derived from the deformation of the MANO mesh.
  • Two-Stage Training: The object is optimized first, followed by the deformable hands, ensuring a coherent joint representation.

3. Key Contributions

1. Fast, Category-Agnostic Framework: GHOST achieves state-of-the-art (SOTA) performance in 3D reconstruction and 2D rendering while running >13× faster than previous category-agnostic methods (e.g., NeRF-based approaches).
2. Prior-Aware Reconstruction: Introduces a strategy to retrieve and align 3D geometric priors from large databases to complete occluded object surfaces.
3. Grasp-Aware Alignment: A novel alignment stage that refines object scale and hand translation specifically during interaction frames to ensure physically realistic contact.
4. Hand-Aware Background Loss: A specialized loss function that distinguishes between true background and object regions occluded by hands, preventing the deletion of valid geometry.
5. Animatable Avatars: Produces fully animatable hand avatars and photorealistic novel-view renderings.

4. Experimental Results

The method was evaluated on ARCTIC, HO3D, and in-the-wild datasets.

• 3D Reconstruction Accuracy:
  • GHOST achieves the lowest Chamfer Distance (CD) metrics for hand-object interaction ($CD_h$) compared to HOLD and BIGS.
  • It significantly reduces hand reconstruction errors (MPJPE) compared to baselines, attributed to the jitter detection and HO alignment steps.
• 2D Rendering Quality:
  • Achieves superior PSNR, SSIM, and LPIPS scores on the ARCTIC dataset compared to HOLD and BIGS.
  • Qualitative results show cleaner, sharper renderings with fewer artifacts (noise/blur) than NeRF-based methods.
• Efficiency:
  • Runtime: ~1 hour per sequence (300 frames) on a single NVIDIA RTX A6000, compared to 13–16 hours for HOLD and BIGS.

5. Significance and Impact

GHOST represents a significant leap forward in hand-object interaction modeling by bridging the gap between physical plausibility and computational efficiency.

• Practicality: The order-of-magnitude speedup makes real-time or near-real-time applications (teleoperation, interactive AR/VR) feasible.
• Robustness: By leveraging geometric priors and grasp-aware reasoning, it solves the "occlusion problem" that has plagued previous category-agnostic methods, allowing for the reconstruction of complete object surfaces even when heavily hidden by hands.
• Generalization: The framework operates without specific object templates, making it applicable to a vast array of real-world scenarios involving novel objects.

The code is publicly available, establishing a new benchmark for fast, realistic, and animatable hand-object reconstruction.