Technical Summary: StableHand

Problem Statement

Recovering the world-space 4D motion of two interacting hands from egocentric video is a critical capability for supervising robot policy learning, where wrist trajectories guide end-effectors and finger articulations define grasp poses. However, this task faces two primary challenges in the egocentric setting:

1. Extended Out-of-View Spans: Coupled head and hand motion often causes one or both hands to exit the camera frustum for prolonged intervals.
2. Persistent Occlusions: Dexterous bimanual manipulation and hand-object interactions lead to severe, often asymmetric, occlusions of the hands.

Existing methods suffer from significant limitations in these scenarios. SLAM-based pipelines transform per-frame camera-frame poses to the world frame without temporal priors, leading to catastrophic drift during missing-hand spans. Native world-space methods learn sequence-level motion priors but typically condition uniformly on visual features, failing to account for the heterogeneity of observation reliability. Specifically, they ignore that wrist global drift errors are significantly larger than finger articulation errors and that occlusions degrade the two hands asymmetrically.

Methodology

The authors propose StableHand, a quality-aware flow-matching framework designed to synthesize world-space dual-hand trajectories by explicitly leveraging the per-component quality of observations. The framework consists of two learned modules trained in sequence:

1. Adaptive Quality Signal and Network

The core insight is that observation quality is heterogeneous across components (wrist vs. fingers) and hands (left vs. right).

• Quality Definition: The system decomposes hand motion quality into four channels: left wrist ($L_W$), left fingers ($L_F$), right wrist ($R_W$), and right fingers ($R_F$). Quality $q \in [0, 1]$ is derived from the error between observed and ground-truth joints using a Radial Basis Function (RBF) kernel: $q = \exp(-e/\sigma)$.
• Quality Network (QN): Since ground-truth errors are unavailable at inference, a learned Quality Network predicts the per-component error $\hat{e}$ (in millimeters) directly. The network takes as input:
  • World-space MANO observations ($\bar{y}$) from a frozen off-the-shelf estimator.
  • Binary hand-confidence flags ($\delta$).
  • Per-frame camera pose ($g$).
  • A learnable trajectory-level context token aggregated via temporal self-attention.
• Training: The QN is pretrained on eight egocentric bimanual corpora (~10M frames). It is supervised by a log-space MSE loss to match error magnitudes and an auxiliary 1D Wasserstein-1 distance loss to align the empirical distribution of predicted errors with ground truth, ensuring the deployment-time bandwidth $\hat{\sigma}$ matches the training-time $\sigma$.

2. Quality-Aware Flow-Matching Generative Model

The generative model is a Diffusion Transformer (DiT) conditioned on the predicted quality signal $\hat{q}$. It employs four coupled mechanisms to handle high-quality and low-quality channels differently:

• Per-Channel Forward Schedule: Unlike standard flow matching which uses a scalar schedule, StableHand uses a channel-specific schedule $\alpha(\tau, \tilde{q})$. For high-quality channels ($q \approx 1$), the schedule freezes the observation ($\alpha \equiv 1$), anchoring the trajectory. For low-quality channels ($q \approx 0$), the schedule treats them as noise ($\alpha \approx 0$), requiring regeneration from the prior.
• Quality-Adjusted Velocity Target: The training objective regresses a velocity target $u_\tau = (1 - \tilde{q}) \odot (x_1 - \varepsilon)$. This ensures the model learns to leave high-quality channels untouched while reconstructing low-quality ones.
• AdaLN Modulation: The DiT denoiser uses Adaptive Layer Normalization (AdaLN) driven by the quality signal to scale updates for wrist and finger tokens separately, matching the per-channel quality structure.
• Quality-Aware Initialization: At inference, the reverse ODE is initialized at $\tau=0$ with $x_0 = \tilde{q} \odot \bar{y} + (1 - \tilde{q}) \odot \varepsilon$. This ensures high-quality channels start near the observation and low-quality channels start near pure noise, eliminating the train-test mismatch found in truncated reverse processes (e.g., SDEdit).

Key Contributions

1. Per-Component Quality Formulation: The paper frames egocentric world-space dual-hand recovery as a per-component, quality-aware generation problem, explicitly identifying the two-axis (component, hand) heterogeneity of observation quality as a structure prior previously left implicit.
2. Quality-Aware Flow-Matching Pathway: A novel generative pathway driven by a four-value quality signal, integrating a per-channel forward schedule, a quality-adjusted velocity target, AdaLN modulation, and a quality-aware ODE initialization.
3. Calibrated Quality Signal: The introduction of a corpus-adaptive quality signal predicted at inference from visual streams, which decouples wrist global drift from finger articulation error and handles asymmetric occlusions.
4. State-of-the-Art Performance: The method achieves superior results on benchmarks featuring long missing-hand spans and persistent occlusions.

Experimental Results

The authors evaluate StableHand on HOT3D (characterized by long missing-hand spans) and ARCTIC (characterized by persistent hand-object occlusion and asymmetric quality).

• HOT3D: StableHand reduces W-MPJPE (World-space Mean Per-Joint Position Error) by 20–25% compared to the strongest baseline (HaWoR), achieving a W-MPJPE of 57.83 mm versus 71.89 mm. The improvement is most pronounced in clips with high missing-hand fractions.
• ARCTIC: StableHand reduces W-MPJPE by ~25% over the strongest baseline, achieving 124.59 mm compared to 171.61 mm. It also shows significant gains in MRRPE (Mean Relative Root Position Error), indicating better joint reasoning about the relative spatial consistency of the two hands.
• Ablation Studies: Removing the per-channel quality schedule causes W-MPJPE to degrade by 1.6x. Replacing the four-token split (wrist/fingers per hand) with a single token degrades performance by 2.1x, confirming the necessity of per-channel modulation. Using a constant or binary quality signal instead of the predicted graded signal results in performance comparable to or worse than baselines.

Significance and Claims

The paper claims that accurate world-space hand motion estimation is tightly coupled with the quality of per-frame observations. By explicitly modeling this quality heterogeneity, StableHand can "anchor" reliable observations while "regenerating" unreliable ones using a learned bimanual motion prior.

The authors position this work as a fundamental step for supervising robot policy learning from large-scale egocentric video, particularly for dexterous and bimanual manipulation tasks where existing methods fail due to drift and occlusion. The framework is noted to be robust across different upstream pose estimators (e.g., WiLoR vs. HaMeR) provided the quality network is recalibrated or retrained.

The paper modestly acknowledges limitations: the quality signal is calibrated to a specific upstream estimator, and in cases of extremely long low-quality horizons where no hand is observed, the model may drift toward plausible but not faithfully grounded configurations. Future work is suggested to include joint object-hand modeling and end-to-end training.