Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves

1. Problem Statement

Understanding Hand-Object Interaction (HOI) is critical for computer vision, robotics, and AR/VR. However, existing data collection methods face significant limitations:

• Lack of Physical Grounding: Vision-only datasets often lack essential physical data like contact forces and motion signals.
• Occlusion: Single-view or limited camera setups suffer from frequent hand occlusions, making robust tracking difficult.
• Domain Gap: While sensing gloves (equipped with IMUs and tactile sensors) provide rich physical data, they create a large visual appearance gap compared to bare human hands. Models trained on glove data fail to generalize to bare-hand tasks due to texture and shape differences.
• Rigid Object Bias: Many existing HOI datasets are limited to pre-scanned rigid objects, failing to capture interactions with non-rigid or deformable objects.

The core challenge is to translate videos of hands wearing sensing gloves into photorealistic bare-hand videos while preserving the synchronized multi-modal sensor signals (tactile and IMU) and handling complex, non-rigid interactions.

2. Methodology: Glove2Hand Framework

The authors propose Glove2Hand, a generative framework that bridges the gap between sensor glove videos and bare-hand videos. The pipeline consists of two main stages:

A. 3D Gaussian Hand Model (Consistency & Geometry)

To ensure temporal consistency and geometric accuracy, the authors introduce a novel 3D Gaussian Hand Model:

• Surface-Grounded Gaussians: Instead of placing 3D Gaussians freely in space (which can break surface integrity during deformation), Gaussians are defined directly on the surface of a learnable canonical hand mesh using barycentric coordinates.
• Deformation Handling: When the hand pose changes, the underlying mesh deforms, and the Gaussians are recomputed on the deformed surface. This leverages the mesh's pre-existing bone weights, avoiding the need to learn skinning weights for individual Gaussians.
• Relightable Rendering: To handle dynamic lighting in egocentric videos, the model predicts environment map coefficients (Spherical Harmonics) using a small MLP conditioned on the hand pose. Crucially, because the surface normals are derived from the mesh geometry (not the Gaussians), the model mitigates the albedo-illumination ambiguity, allowing for robust relighting.

B. Diffusion Hand Restorer (Generative Refinement)

The rendered 3D Gaussian hand is a "hand-only" sequence. To integrate it seamlessly into the scene and handle complex interactions:

• Input Preparation: The rendered hand is overlaid onto the original glove video. The glove region is erased using an optical-flow-based inpainter (Propainter), and the wrist region is masked to remove the glove-arm connection.
• Diffusion Model: A ControlNet-based diffusion model (trained on bare-hand videos) acts as a "restorer." It takes the corrupted input (rendered hand + masked background) and generates a photorealistic bare hand that seamlessly blends with the background, refines skin textures, corrects wrist connections, and resolves complex hand-object boundaries (e.g., occlusion, contact).

C. Inference Pipeline

1. Extract hand poses from the input glove video.
2. Render a consistent, hand-only video using the 3D Gaussian model.
3. Remove the glove from the original video using inpainting (preserving foreground objects).
4. Overlay the rendered hand and feed the result into the Diffusion Restorer to generate the final photorealistic bare-hand video.

3. Key Contributions

1. Glove2Hand Framework: The first method to generate photorealistic, temporally consistent bare-hand videos from sensor glove captures, bridging the visual domain gap while preserving physical sensor data.
2. Novel 3D Gaussian Hand Model: A minimalist, surface-grounded Gaussian representation that ensures geometric consistency and enables dynamic relighting without heavy multi-view setups.
3. HandSense Dataset: The creation of the first multi-modal HOI dataset featuring:
   • Synchronized glove-to-hand video pairs.
   • Continuous, sensor-measured tactile signals (ground truth for contact).
   • IMU signals (ground truth for pose under occlusion).
   • Interactions with diverse objects, including non-rigid and deformable items.
4. Downstream Applications: Demonstrating that the synthesized data significantly improves performance in:
   • Vision-based Contact Estimation: Using tactile signals as annotation-free ground truth.
   • Robust Hand Tracking: Using IMU signals as pseudo-ground-truth for tracking under severe occlusion.

4. Experimental Results

The authors evaluated Glove2Hand on the HandSense dataset and compared it against state-of-the-art video inpainting and generative models (e.g., Pix2Pix, BrushNet, HandRefiner).

• Quantitative Metrics: Glove2Hand achieved the best performance in FID (30.1) and FVD (19.5), significantly outperforming baselines. This indicates superior image quality and temporal consistency.
• Human Evaluation: In a user study, synthesized videos were rated nearly indistinguishable from real videos, with a perceptual gap of only 0.51 (on a 1.0 scale) for hand realism in controlled settings.
• Ablation Studies: Removing any component (Surface Grounding, Relighting, Diffusion, Glove Removal) resulted in significant performance drops, confirming the necessity of the full pipeline.
• Downstream Task Performance:
  • Contact Estimation: Training on Glove2Hand data improved Contact IoU from 71.5% (glove-only) to 88.2% (combined real + G2H), surpassing glove-only training.
  • Hand Tracking: Fine-tuning a tracker (UmeTrack) with Glove2Hand data reduced Mean Keypoint Position Error (MKPE) in occluded scenarios from 19.2mm to 16.6mm, whereas training on raw glove data degraded performance due to the domain gap.

5. Significance

• Data Curation Engine: Glove2Hand enables the creation of large-scale, physically grounded HOI datasets without the prohibitive cost of manual annotation or multi-camera studios.
• Physical Grounding: By preserving tactile and IMU signals, the framework allows researchers to train vision models that understand not just what is happening visually, but how forces and contacts are applied.
• Robustness: The ability to synthesize data with IMU-grounded poses under occlusion solves a major bottleneck in training robust hand trackers for real-world, cluttered environments.
• Generalization: The framework successfully handles non-rigid objects and dynamic lighting, moving beyond the limitations of rigid-object-only datasets.

In conclusion, Glove2Hand represents a paradigm shift in HOI data collection, transforming sensor-rich but visually limited glove data into high-fidelity visual data that drives advancements in computer vision and robotics.