💻 computer science

Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch

This paper proposes a multimodal, physically grounded 3D generative framework that leverages proprioception and multi-contact touch signals to achieve metric-scale amodal object reconstruction and pose estimation under severe hand occlusion, significantly outperforming vision-only baselines by reducing ambiguity and ensuring physical consistency through physics-based objectives and differentiable decoder guidance.

Original authors: Gabriele Mario Caddeo, Pasquale Marra, Lorenzo Natale

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gabriele Mario Caddeo, Pasquale Marra, Lorenzo Natale

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what a mysterious object looks like while it's being held tightly in your hand. You can only see a tiny sliver of it peeking out from between your fingers. If you were just using your eyes (vision), you'd be guessing wildly. You might imagine a ball when it's actually a cube, or you might guess the wrong size entirely.

This paper presents a new way for robots to "see" objects in these tricky situations. Instead of just relying on their cameras, the robot uses three senses at once: sight, the feeling of its own hand position (proprioception), and the actual feeling of touch.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Blindfolded Sculptor"

Imagine a sculptor trying to carve a statue, but they are wearing a blindfold and only allowed to peek through a tiny hole in a curtain. They can see a little bit of the statue, but most of it is hidden behind their own hands.

Old methods (Vision-only): The sculptor tries to guess the rest of the statue based only on that tiny peek. They might guess the statue is floating in mid-air or that their hands are passing right through the stone (which is physically impossible).
The new method: The sculptor stops guessing blindly. They use their sense of touch and the feeling of their own arm position to figure out exactly where the statue must be.

2. The Three Clues (The "Super-Senses")

The robot combines three types of information to build a perfect 3D model:

The Eye (Egocentric RGB Image): This is the camera view. It sees the parts of the object that aren't covered by the hand.
The Muscle Memory (Proprioception): The robot knows exactly where its fingers are because it has sensors in its joints. It knows, "My thumb is here, my pinky is there." This acts like a rigid frame that the object cannot pass through.
The Touch (Multi-Contact Touch): The robot has sensitive skin on its fingertips. When it touches the object, it gets a "ping" saying, "Hey, I'm touching something right here!" This tells the robot exactly where the surface of the object is, even if the camera can't see it.

3. The Magic Process: "The Physics-First Generator"

The robot uses a smart computer program (an AI) to put these clues together. Think of this AI as a 3D printer that follows the laws of physics.

Step A: The Rough Draft (The SDF):
The AI first creates a "cloud" of data representing the object's shape. It's like a digital fog. The AI knows the object is somewhere in this fog.
- The Physics Trick: The AI is programmed with two golden rules:
  1. No Ghost Hands: The object cannot be inside the robot's hand (no interpenetration).
  2. Touch the Touch: The surface of the object must touch the spots where the robot's fingers felt a bump.
    By forcing the AI to follow these rules, it eliminates the wild guesses. It can't draw a shape that floats through the hand or misses the touch points.
Step B: The Polish (Refinement):
Once the rough shape is locked in by the physics rules, a second step adds the fine details and textures, making it look like a real, high-quality 3D model.

4. Why This is a Big Deal

It's "Physically Grounded": Previous AI models were like artists who only cared if the picture looked pretty. If the picture showed a hand going through a cup, they didn't care. This new model is like an engineer; if the hand goes through the cup, the model says, "Wait, that's impossible!" and fixes it.
It Works in the Real World: The researchers tested this on a real humanoid robot. Even though the robot's hand was different from the one the computer learned on, the system still worked. It's like learning to drive a car and then being able to drive a truck without needing to relearn everything from scratch.

The Bottom Line

This paper teaches robots to stop guessing when they can't see the whole picture. By combining what they see, where their hand is, and what they feel, they can reconstruct hidden objects with perfect accuracy, ensuring the robot doesn't try to grab a ghost or crush a fragile item. It's the difference between guessing what's in a wrapped gift and actually feeling the shape of the box through the wrapping paper.

1. Problem Statement

Robotic manipulation requires accurate 3D geometry of objects to plan stable grasps, avoid collisions, and execute in-hand manipulation. However, during grasping and manipulation, the robot's hand severely occludes the object, rendering single-view RGB reconstruction fundamentally under-constrained.

Limitations of Vision-Only: Existing 3D generative models (e.g., diffusion-based) often produce visually plausible but physically invalid reconstructions under heavy occlusion. They may generate shapes that intersect the hand (interpenetration), miss true contact regions, or drift in metric scale, making them unusable for control.
The Gap: While tactile sensing has been used for grasping, it is rarely integrated as an explicit geometric constraint for dense 3D reconstruction in occluded scenarios. There is a need for a method that fuses vision with physical interaction signals (proprioception and touch) to resolve occlusion ambiguities.

2. Methodology

The authors propose a multimodal, two-stage generative pipeline that reconstructs metric-scale object geometry by fusing egocentric RGB images, hand proprioception, and multi-contact tactile feedback.

Core Representation

Pose-Aware, Camera-Aligned SDF: The object geometry is represented as a Signed Distance Field (SDF) within a 3D grid. Crucially, this grid is camera-aligned (the grid's $z$ -axis matches the camera viewing direction) and pose-aware, allowing the object and the hand to share the same canonical 3D domain. This enables direct physical constraints between the hand and object.
Structured Latents (SLat): A Variational Autoencoder (Structure-VAE) learns a compact latent space for SDF grids, enabling efficient generation.

Pipeline Architecture

The system operates in two stages (illustrated in Fig. 1 and Fig. 2):

Stage A: Coarse Structure Generation (Flow Matching)

Encoder: A Structure-VAE encoder compresses ground-truth SDF grids into latent vectors.
Pre-training: A conditional Flow Transformer is trained on vision-only data (clean object images) to learn a strong shape prior.
Fine-tuning: The model is fine-tuned on simulated grasp scenes with heavy occlusion. It conditions on:
- Visible RGB Evidence: The visible part of the object and occlusion masks.
- Proprioception: The hand mesh (derived from joint angles) is rasterized into an SDF and encoded into a latent vector.
- Tactile Feedback: Multi-contact touch is represented as a 3D tensor (contact occupancy + distance field) and fused with the hand latent.
Physics-Guided Losses: During fine-tuning, the model minimizes:
- Non-Interpenetration Loss: Penalizes object mass inside the hand volume.
- Contact-Consistency Loss: Pulls the reconstructed surface toward observed tactile contact points.

Stage B: Refinement and Texturing

The coarse SDF from Stage A is passed to a refinement module (adapted from SLat/Sam3D) to generate a high-fidelity, textured mesh. This stage maintains the pose and occlusion consistency established in Stage A.

Inference with Physics Guidance

During sampling, the authors introduce a decoder-guided physics guidance mechanism. At each denoising step, the current latent is decoded to an SDF, and a control term is added to the velocity field based on the gradient of the physical energy (interpenetration and contact losses). This steers the generation toward physically feasible shapes in real-time.

3. Key Contributions

Physically Grounded Generative Inference: A novel method to inject physical constraints (contact and non-interpenetration) directly into a generative 3D reconstruction pipeline, moving beyond purely visual plausibility.
Multimodal Conditioning in Shared 3D Space: A pose-aware, camera-aligned SDF representation that aligns 2D visual evidence, 3D hand pose (proprioception), and tactile contacts within a single 3D grid, enabling direct geometric reasoning.
Robustness to Occlusion: Demonstrates that integrating proprioception and touch significantly improves amodal completion under severe occlusion compared to vision-only baselines.
Cross-Embodiment Transfer: Validates the approach on a real humanoid robot with an end-effector different from those used in training, proving generalization across different robotic hands.

4. Experimental Results

The method was evaluated in simulation and on a real humanoid robot (Italian Institute of Technology).

Datasets: Training used 3D-FUTURE, HSSD, ABO, and ObjaverseXL. Fine-tuning used Google Scanned Objects (GSO). Testing used the YCB dataset.
Baselines: Compared against Amodal3R (vision-only amodal completion) and SAM3D (state-of-the-art single-image 3D reconstruction).
Simulation Results:
- Reconstruction Quality: The proposed method outperformed baselines in Chamfer Distance (CD), Normal Consistency (NC), and Voxel IoU, especially in high-occlusion bins (B4, B5).
- Pose Estimation: Achieved significantly better 3D IoU and lower rotation error (ICP-Rot) than SAM3D, as proprioception directly constrains the hand-object configuration.
- Ablation: Removing touch degraded performance in heavy occlusion; removing proprioception caused significant ambiguity.
Real-World Results:
- Deployed on a humanoid robot with XELA tactile sensors and a different hand morphology than the training data.
- The method maintained high reconstruction quality and physical plausibility, whereas vision-only baselines (Amodal3R) failed significantly under real-world noise.
- Limitations: Performance is sensitive to calibration errors (camera-hand extrinsics) and forward kinematics accuracy.

5. Significance

This work bridges the gap between Perception and Physical Interaction in robotics. By treating 3D reconstruction as a physically grounded generative problem, the authors demonstrate that robots can "feel" their way to a better understanding of object geometry when vision fails.

Practical Impact: The output is a metric-scale, physically consistent mesh that can be immediately used for downstream tasks like collision avoidance, insertion, and force control.
Scientific Contribution: It establishes a new paradigm where tactile and proprioceptive signals are not just used for control but are fundamental inputs for generating the 3D world model itself, solving the "under-constrained" nature of occluded vision.