Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priors

Technical Summary: Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priors

Problem Statement
Monocular 3D hand reconstruction remains a challenging, under-constrained problem. While high-performing models exist, they often produce geometric predictions that do not perfectly align with the observed image appearance. The authors observe that current training signals, while effective, fail to fully exploit available appearance information. Furthermore, existing datasets with dense texture supervision are typically limited to multi-view studio conditions, lacking the diversity and scalability of "in-the-wild" monocular data. The core challenge is to leverage sparse, partial, and noisy texture observations from single images to improve both the geometric accuracy and the realism of 3D hand reconstructions without requiring ground-truth textures or multi-view inputs.

Methodology
The proposed approach integrates a lightweight, learnable texture module into an existing geometry-driven reconstruction pipeline (specifically augmenting the HaMeR transformer architecture). The method operates through three primary components:

1. Sparse Observation Extraction: Given an input image and a predicted 3D hand mesh (from a base model like HaMeR), the system back-projects visible surface points to extract sparse UV-RGB pairs. These pairs represent partial, view-specific texture observations.
2. Learned Texture Module: A transformer-based architecture processes these variable-length, sparse observations.
   • Encoder: RGB values are embedded via a 1×1 convolution, while UV coordinates are encoded using Random Fourier Features to inject spatial context. These are concatenated and processed by a Transformer decoder using learned query tokens to aggregate global features.
   • Decoder: A convolutional upscaling decoder (utilizing pixel shuffling) converts the latent representation into a dense, full-resolution UV texture map (224×224).
   • Training Strategy: The module is trained in a weakly-supervised manner. It reconstructs a full texture map that agrees with observed colors at their corresponding UV positions. The loss function combines a masked L1 term with a Fourier-domain consistency loss to ensure both local accuracy and global frequency coherence.
3. Dense Alignment Supervision: The reconstructed texture is applied to the predicted mesh and rendered back into the image space using a differentiable renderer (PyTorch3D). A photometric consistency loss is computed between this rendered image and the original input. This dense alignment signal guides the refinement of the base model's geometric estimates.

Crucially, the texture module acts as a "sidecar" during training. In the proposed configuration, the backbone of the base model is frozen, and only the head is fine-tuned alongside the texture module (which is pre-trained or "warmed up"). At inference time, the texture module is not required; the improvement manifests solely through the updated weights of the reconstruction model.

Key Contributions

• Unified Sparse-to-Dense Texture Model: The authors introduce the first method to accommodate sparse, partial hand texture observations into a unified model for full texture reconstruction. This is achieved via weak supervision without requiring ground-truth textures or multi-view data.
• Transformer-Based Architecture: A novel architecture employing pixel-level attention and Random Fourier Features is designed to reconstruct dense, coherent textures from incomplete UV-RGB inputs.
• Scalable Training Procedure: A warm-up procedure is proposed that initializes the model using a mix of synthetic and real-world data, eliminating the need for manual annotations.
• End-to-End Photometric Supervision: The integration of a differentiable rendering framework with photometric consistency losses allows for texture-guided supervision that improves geometric alignment without manual intervention.

Results
The method was evaluated by augmenting HaMeR [46] and tested on the HInt dataset (comprising NEWDAYS, VISOR, and Ego4D) as well as standard benchmarks like FreiHAND and HO3D.

• Quantitative Improvements: The approach consistently outperforms the baseline HaMeR, particularly in "Occluded" keypoint categories. For instance, on the VISOR dataset at the @0.10 threshold, the method achieved 64.2% PCK for occluded keypoints compared to 61.8% for HaMeR. Improvements were also observed on NEWDAYS and Ego4D.
• Benchmark Performance: On FreiHAND and HO3D, the method achieved performance comparable to the baseline, demonstrating that the texture-guided supervision does not degrade performance on clean, studio-captured datasets while providing gains in challenging, occluded, or egocentric scenarios.
• Robustness to Sparsity: Experiments showed that the texture model maintains high reconstruction fidelity with as few as 1,000 visible UV pixels, indicating robustness to the sparse supervision typical of monocular data.
• Efficiency: The texture module adds approximately 70ms per training iteration and introduces zero computational cost at test time, as the final model relies only on the updated weights of the base network.

Significance and Claims
The paper claims that appearance-guided alignment serves as a meaningful complement to geometry-driven training, particularly in scenarios where geometric cues are ambiguous (e.g., severe occlusion or egocentric viewpoints). The authors emphasize that their solution is practical and modular: it requires no extra manual annotation, scales to uncurated monocular data, and improves both geometric accuracy and visual realism. The work establishes a foundation for future improvements by demonstrating that learned texture priors can be effectively integrated into existing reconstruction pipelines to resolve ambiguities in 3D hand pose and shape estimation.