Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation

1. Problem Statement

The paper addresses the challenge of 3D hand pose estimation from monocular RGB images. While critical for applications like AR/VR, human-computer interaction, and sign language recognition, this task remains difficult due to:

• Severe self-occlusions and object occlusions.
• Large articulation ranges and foreshortening.
• Depth ambiguity in 2D-to-3D projection.

Existing methods often rely on the MANO parametric model and geometric constraints (e.g., bone length consistency) but frequently fail to leverage the semantic regularities of hand gestures. In domains like sign language, specific hand shapes correspond to lexical units, offering recurring sub-configurations that could aid in disambiguating poses under occlusion. The authors hypothesize that integrating these semantic priors can serve as a powerful inductive bias to improve estimation accuracy.

2. Methodology

The proposed framework is a two-stage approach that couples gesture-aware pretraining with gesture-guided token fusion.

Stage 1: Gesture-Aware Pretraining

• Backbone: The authors use HRNet (High-Resolution Network), which maintains high-resolution feature maps throughout the network, making it suitable for dense prediction tasks.
• Task: The HRNet encoder is pretrained on the InterHand2.6M dataset using a hierarchical gesture classification task.
• Label Construction: The authors manually curated 54 "coarse" and 70 "fine" gesture categories from the dataset. Visually identical gestures share labels, while subtle variations (e.g., thumb position) are assigned distinct fine labels under a common coarse category.
• Architecture:
  • Global features ($g$) are extracted from the F5 feature map.
  • Two lightweight classification heads predict coarse ($\gamma_{coarse}$) and fine ($\gamma_{fine}$) gesture logits.
  • Training Strategy: A combined cross-entropy loss is used with a gradual weighting schedule. The weight for the fine-grained loss ($\alpha$) starts low and increases over epochs, allowing the model to first learn coarse semantic separation before refining fine-grained distinctions.

Stage 2: Gesture-Guided 3D Pose Estimation

This stage refines the pose estimation using the semantic embeddings learned in Stage 1.

• Per-Joint Tokenization (2.5D Volumetric Reasoning):
  • Multi-scale features (F4, F5) are concatenated and flattened into a global token sequence.
  • A volumetric head predicts a 2.5D heatmap volume ($H$) for each of the 21 joints. The $z$-axis represents relative depth (wrist-relative) rather than absolute metric depth.
  • Joint locations are estimated via soft-argmax over this volume.
  • Per-joint tokens ($U$) are sampled from the spatial feature map using the estimated 2D coordinates.
• Gesture-Guided Transformer:
  • The gesture logits from Stage 1 are projected into embedding space ($e_{coarse}, e_{fine}$).
  • These embeddings are combined with learnable type embeddings to form gesture guidance tokens ($c^*$).
  • A Transformer encoder processes the concatenation of per-joint tokens ($Q$) and gesture guidance tokens ($S = [Q; c_{coarse}; c_{fine}]$).
  • The Transformer refines the joint tokens, using gesture semantics as intermediate representations to guide the reasoning process.
• Regression: The refined tokens are decoded via MLPs to predict MANO parameters:
  • Pose: 6D joint rotations (for continuity).
  • Shape: 10 shape coefficients ($\beta$).

Training Objective

The total loss function is a weighted sum of multiple supervision levels:

• Parameter Level: $\ell_1$ loss on MANO pose and shape parameters.
• Joint Level: $\ell_1$ losses on 3D joint positions (from MANO kinematics), 2.5D heatmap coordinates, and 2D reprojection errors.
• Structural: A continuity loss ($L_{cont}$) penalizing discontinuities in consecutive joint rotations to ensure anatomically smooth poses.

3. Key Contributions

1. Gesture-Aware Pretraining Strategy: A novel method to equip high-resolution encoders with coarse-to-fine gesture semantics, motivated by the link between hand pose and linguistic meaning in sign language.
2. Gesture-Guided Fusion Framework: A mechanism that uses gesture embeddings as intermediate representations within a Transformer to guide per-joint token refinement, effectively injecting semantic priors into the regression pipeline.
3. Architecture-Agnostic Transferability: Demonstrated that the gesture-pretrained backbone can be plugged into existing architectures (specifically EANet) without modification, consistently reducing error rates.

4. Experimental Results

Experiments were conducted on the single-hand subset of InterHand2.6M.

• Performance Metrics: The authors report Mean Per Joint Position Error (MPJPE) and Mean Per Vertex Position Error (MPVPE).
• Comparison with SOTA:
  • EANet (Baseline): 4.94 mm MPJPE.
  • Ours (Full Model): 4.84 mm MPJPE and 5.19 mm MPVPE.
  • Cross-Architecture Transfer: Replacing EANet's ImageNet-pretrained ResNet-50 with the authors' gesture-pretrained HRNet backbone (without other changes) reduced EANet's error from 4.94 mm to 4.86 mm.
• Qualitative Analysis:
  • t-SNE Visualization: Pretrained features showed significantly better inter-class separation compared to the baseline, confirming the encoder learned discriminative semantic features.
  • Occlusion Handling: Under hand-object and self-occlusion, the proposed method produced more anatomically plausible meshes with fewer interpenetration artifacts compared to the baseline.

5. Significance

This work bridges the gap between gesture semantics and 3D pose estimation. By treating gesture labels not just as classification targets but as inductive biases, the authors demonstrate that semantic understanding of hand configurations can resolve ambiguities caused by occlusion and depth uncertainty.

The key takeaway is that semantic priors are transferable. The benefits of gesture-aware pretraining are not limited to the specific architecture used for training; they generalize to other state-of-the-art models (like EANet), suggesting a generalizable component for future 3D hand pose estimation systems. The paper also highlights the potential for extending this approach to two-hand interactions and "in-the-wild" scenarios.