Technical Summary: ScaleHP – Estimating Hand Pose in Metric Space

1. Problem Definition

Accurate hand pose estimation (HPE) in metric space (absolute camera coordinates with real-world scale) is critical for immersive human-computer interaction (HCI), virtual/augmented reality (VR/AR), and robotics. However, existing methods predominantly operate in root-relative coordinate systems, predicting hand poses relative to a reference joint (e.g., the wrist) without determining the absolute size or distance of the hand from the camera.

The core challenge in transitioning to metric space is the inherent depth-scale ambiguity of monocular images. Current approaches attempting to resolve this often rely on:

• Auxiliary depth estimation modules: These are fragile and fail when the background deviates from training distributions or when only the hand is visible.
• Parametric model regression: These often ignore absolute scale or require pre-computed depth maps.
• Two-stage pipelines: Estimating root-relative pose first, then lifting to camera space using external scale parameters or assumptions.

These methods struggle to provide stable, precise metric predictions in generalized, real-world scenarios.

2. Methodology: ScaleHP

The authors propose ScaleHP, an end-to-end, one-stage framework that bypasses fragile extrinsic depth modules by leveraging intrinsic anthropometric priors. The core hypothesis is that the proportional relationships among human hand bones encode stable information about the hand's overall metric size, which can be inferred directly from the image.

2.1 Architecture Overview

ScaleHP consists of three main components (illustrated in Fig. 2):

1. Frozen Detector Backbone: Utilizes a pre-trained, frozen DETR-style detector (e.g., Grounding DINO) to extract multi-modal features and localize hands via text prompts. This provides robust feature extraction without training the backbone.
2. Metric 2D–3D Decoder: A novel Transformer-based decoder that lifts 2D observations to 3D metric space.
3. Training-Free Analytical Module: A geometric solver that computes the final translation vector based on perspective constraints.

2.2 Key Components

The Scale Token

The central innovation is the introduction of a dedicated scale token ($Q_s$) within the Transformer decoder.

• Initialization: Initialized from the hand instance query.
• Interaction: The scale token engages in self-attention with 2D and 3D joint queries to learn skeletal proportions and topology. It then interacts with multi-scale image features via deformable attention to capture global appearance and spatial occupancy.
• Prediction: The token outputs a scalar global scale ($s$), representing the mean length of specific hand bones (e.g., index, middle, ring fingers). This allows the model to learn a deterministic mapping from pixel configurations to real-world lengths.

Metric Recovery via Perspective Constraints

Once the global scale $s$ is predicted, the framework recovers the absolute translation vector $t$ without learning a separate depth regressor.

• Formulation: The relationship between root-relative 3D coordinates ($J_{rel}$), 2D pixel coordinates ($J_{2D}$), and camera intrinsics ($K$) is modeled under perspective projection.
• Solution: The authors formulate a system of linear equations (2N equations for N keypoints) where the unknown is the depth translation ($t_z$). This system is solved using a least-squares approach.
• Result: The recovered $t$ and predicted $s$ are applied to the root-relative pose to reconstruct the hand in the true metric camera coordinate system.

2.3 Loss Functions

The model is trained end-to-end using four supervision signals:

1. 2D Supervision: L1 loss and Object Keypoint Similarity (OKS) on 2D keypoints.
2. 3D Supervision: L1 loss on root-relative 3D keypoints.
3. Metric Scale Supervision: Mean Squared Error (MSE) between the predicted scale scalar and the ground-truth scale derived from bone lengths.
4. Projection Supervision: Weak supervision by projecting predicted 3D joints back to 2D and comparing with annotations.

3. Key Contributions

1. Metric-Aware Framework: The paper introduces a framework that explicitly models the global scale of the hand using intrinsic anatomical bone proportions, mitigating depth ambiguity without relying on external depth estimators.
2. ScaleHP Architecture: The first one-stage end-to-end framework for metric-space HPE, featuring a scale token that interacts with pose queries to learn a direct mapping from image features to absolute physical scale.
3. Training-Free Analytical Solver: A novel approach to recovering global translation via a least-squares solution under perspective constraints, avoiding the need for learned depth regressors.
4. State-of-the-Art Performance: The method achieves superior results on standard benchmarks, demonstrating that internal biological constraints significantly reduce both relative geometry and absolute metric errors.

4. Experimental Results

The authors evaluated ScaleHP on FreiHand, DexYCB, and HO3Dv3 benchmarks.

• Primary Metric (CS-MPJPE): This measures the average Euclidean distance in camera space without alignment.

  • FreiHand: 35.8 mm (vs. 42.4 mm for the next best baseline, NFV).
  • DexYCB: 136.3 mm (vs. 183.2 mm for CMR + GS).
  • HO3Dv3: 50.7 mm (vs. 132.6 mm for HandDGP + GS).
  • Note: The authors note that baselines often require an "Optimal Global Scale" (GS) post-hoc adjustment to be competitive, whereas ScaleHP predicts this scale end-to-end.

• Aligned Metrics (P-MPJPE / R-MPJPE):

  • On FreiHand, ScaleHP achieves a P-MPJPE of 5.0 mm, tying the state-of-the-art.
  • On DexYCB, it achieves 4.6 mm P-MPJPE and 10.3 mm R-MPJPE.
  • On HO3Dv3, it achieves 5.9 mm P-MPJPE.

• Ablation Studies:

  • Scale Token: Removing the scale token increases CS-MPJPE significantly (e.g., from 35.8 to 44.3 mm on FreiHand), confirming its necessity for metric accuracy.
  • Depth Ambiguity: The scale token also improves relative geometry (P-MPJPE), specifically reducing errors along the Z-axis (depth), indicating it acts as a geometric regularizer.
  • Training Strategy: Joint training of the scale token with joint queries outperforms decoupled pre-training/fine-tuning strategies.

5. Significance and Claims

The paper claims that ScaleHP offers a robust solution for generalized, real-world hand tracking by shifting the paradigm from fragile, context-dependent depth estimation to intrinsic biological priors.

• Robustness: By deriving scale solely from the hand's internal structure, the method remains invariant to extrinsic environmental factors (e.g., background changes, scene context), effectively bypassing the scale-depth ambiguity inherent in monocular 3D reconstruction.
• Generalization: The framework demonstrates strong performance on "in-the-wild" images, handling hand-object interactions, severe occlusions, and large viewpoint changes where previous methods often fail.
• Efficiency: The one-stage, end-to-end design avoids the computational overhead and error accumulation of multi-stage pipelines or auxiliary depth networks.

The authors conclude that leveraging anthropometric regularities allows for the reconstruction of hand geometry in absolute camera-coordinate metric space, providing a foundation for applications requiring precise spatial understanding, such as teleoperation and dexterous robotic manipulation.