Fast-HaMeR: Boosting Hand Mesh Reconstruction using Knowledge Distillation

1. Problem Statement

Accurate and real-time 3D hand reconstruction is critical for applications in VR/AR, robotics, and human-computer interaction. However, state-of-the-art (SOTA) methods, particularly those based on large transformer architectures like HaMeR (which uses a ViT-H backbone), are computationally expensive. These heavy models struggle to run in real-time on resource-constrained devices such as smartphones, headsets, and embedded systems. The core challenge is to accelerate these models and reduce their parameter count while maintaining the high reconstruction accuracy required for complex tasks like handling occlusions and diverse hand poses.

2. Methodology

The authors propose a Knowledge Distillation (KD) framework designed to transfer knowledge from a large, high-capacity "Teacher" model to a lightweight "Student" model.

• Teacher Model: The baseline is HaMeR, a SOTA method utilizing a ViT-Huge (ViT-H) backbone. It regresses MANO parameters ($\theta, \beta$) and camera parameters to generate 3D hand meshes and joint locations.
• Student Models: The authors replace the heavy ViT-H backbone with four lightweight alternatives:
  • MobileNet (depthwise convolutions)
  • MobileViT (lightweight hybrid transformer)
  • ConvNeXt (modernized CNN)
  • ResNet (residual connections)
  • These backbones reduce the model size to approximately 35% of the original HaMeR.
• Distillation Strategies: The student network is trained to mimic the teacher at three levels:
  1. Output-Level Distillation: The student learns to match the teacher's final predictions (3D joints $K^{3D}$, 2D joints $K^{2D}$, and MANO parameters) rather than just ground truth.
  2. Feature-Level Distillation: The student learns to replicate the intermediate feature maps of the teacher. A learnable $1\times1$ convolution aligns channel dimensions, and bilinear interpolation aligns spatial dimensions.
  3. Combined Distillation: A hybrid approach utilizing both output and feature losses simultaneously.
• Training Objective: The total loss function combines standard ground-truth supervision ($L_{GT}$) with distillation losses ($L_{KD}$), weighted by hyperparameters $\lambda_{KD}$ and $\gamma_{feat}$.

3. Key Contributions

• Efficient Architecture Replacement: Successfully demonstrated that replacing the ViT-H backbone with lightweight backbones (specifically ConvNeXt-L) can achieve near-SOTA performance with significantly reduced computational cost.
• Distillation Analysis: Systematically evaluated how different student architectures respond to different distillation types:
  • Output-level distillation proved most effective for smaller or structurally aligned models (e.g., MobileViT).
  • Feature-level distillation yielded superior results for higher-capacity students (e.g., ConvNeXt-L), allowing them to better capture the teacher's spatial and semantic cues.
• Performance Benchmarking: Established a new balance between speed and accuracy, achieving 1.5x faster inference with only a 0.4mm drop in accuracy compared to the original HaMeR.

4. Experimental Results

Experiments were conducted on the HO3D-v2 dataset using an RTX 4060 Ti GPU.

• Baseline Performance: Without distillation, lightweight backbones showed a trade-off: MobileViT-S was 1.55x faster but 1.6mm less accurate; ResNet-50 was 1.85x faster but 1.0mm less accurate.
• Distillation Impact:
  • Output-Level: Improved MobileViT-S by 0.4mm at $\lambda_{KD}=0.5$.
  • Feature-Level: The best results were achieved with ConvNeXt-L using feature-level distillation ($\lambda_{KD}=0.8, \gamma_{FD}=12$). This configuration improved accuracy by 0.2mm over its non-distilled baseline.
• Final Metrics (Best Model: ConvNeXt-L + Feature KD):
  • PA-MPJPE (Joint Error): 8.1 mm (vs. 7.7 mm for HaMeR).
  • PA-MPVPE (Vertex Error): 8.3 mm (vs. 7.9 mm for HaMeR).
  • Inference Speed: 40 FPS (vs. 27 FPS for HaMeR), representing a 1.48x speedup.
  • Model Size: Reduced from 671M parameters (HaMeR) to 240M parameters.
• Comparison: The proposed method outperforms other SOTA methods like HandOccNet, METRO, and ArtiBoost in terms of the speed-accuracy trade-off, approaching the performance of the latest Hamba and WiLoR models but with significantly lower latency.

5. Significance

This work bridges the gap between high-fidelity 3D hand reconstruction and real-world deployment on edge devices. By proving that Knowledge Distillation can effectively compress transformer-based 3D reconstruction models, the authors enable:

• Real-time applications on mobile and AR/VR hardware without requiring cloud processing.
• Energy efficiency, crucial for battery-powered devices.
• Scalability, offering a practical pathway to deploy complex 3D vision tasks in resource-constrained environments.

The code and models are publicly available, facilitating further research into efficient 3D vision.