Technical Summary: HAP - A Hand-Driven Active Perception Framework for Egocentric Head Motion Prediction

1. Problem Formulation

The paper addresses the underexplored challenge of egocentric head motion forecasting. While existing forecasting methods predominantly focus on hand trajectories, interaction hotspots, or action sequences, they often treat head motion as a secondary consequence or fail to explicitly model the perceptual demands driving it.

The authors argue that head reorientation is an active perception mechanism: during manipulation, the head actively redirects to acquire task-relevant visual evidence, particularly when targets become occluded. The core problem is formulated as predicting future 6-DoF (six Degree of Freedom) head trajectories conditioned on:

1. Observed hand motion history.
2. Inferred target context (interaction intent).
3. Dynamic visibility and occlusion relationships among scene objects.

The task presents two primary challenges:

• Intent Inference: Determining the intended interaction target from incomplete hand and object observations.
• Dynamic Occlusion Reasoning: Modeling how perceptual demands change as the viewpoint shifts and objects occlude one another, requiring the head to reorient to maintain visual access.

2. Methodology: The HAP Framework

The proposed Hand-Driven Active Perception (HAP) framework predicts future 6-DoF head trajectories by integrating motion dynamics with perceptual reasoning. The pipeline consists of four main components:

A. Hand-Driven Target Intention Estimation

HAP infers the probability distribution of candidate targets based on observed hand states and object geometry.

• Input: Hand position ($h_\tau$), object masks (propagated via SAM 2), and relative geometry features (e.g., distance reduction, directional consistency of hand movement toward an object).
• Mechanism: A fusion network combines object embeddings, hand embeddings, and relative geometry features. Self-attention contextualizes candidates within the frame, followed by a Gated Recurrent Unit (GRU) to accumulate evidence over time.
• Output: A soft probability distribution ($\pi_{\tau, i}$) over candidate targets, preserving uncertainty rather than making a hard assignment.

B. Dynamic Predictive Target-Centric Amodal Occlusion Graph (P-TAOG)

To capture the evolving perceptual demand, HAP constructs a directed graph representing occlusion relationships.

• Construction: Using sparse completed point sets (from SAM 3D Objects) registered to the initial camera frame, the system projects points into the current camera view.
• Occlusion Logic: It calculates a current occlusion ratio (based on z-buffer depth) and a potential occlusion ratio (estimating risk under local image plane perturbations without future pose access).
• Graph Structure: A fully connected directed graph $G_\tau$ where edges $(j, i)$ encode the occlusion influence of object $j$ on object $i$. Edge features include 2D/3D distances, depth differences, and both current/potential occlusion ratios.
• Encoding: The graph is processed via Directed Principal Neighbourhood Aggregation (PNA) layers to separately aggregate incoming and outgoing messages, followed by a causal GRU to encode temporal evolution.

C. Head Motion Prediction and Horizon-Wise Gating

The final prediction stage fuses motion history with the perceptual graph representation.

• Feature Fusion: The base motion feature (hand displacement and head transform) is concatenated with the P-TAOG representation.
• Temporal Encoding: A two-layer GRU encodes the history. For each prediction horizon $h$, a learned query attends to the history and combines it with a constant velocity prior (extrapolated from the last observed transform).
• Horizon-Wise Gate: A learned gate $\alpha_{t,h}$ blends the learned trajectory ($d_{t,h}$) with the constant velocity prior ($c_{t,h}$):
  $$\hat{y}_{t,h} = c_{t,h} + \alpha_{t,h} \odot (d_{t,h} - c_{t,h})$$
  This allows the model to adaptively depart from the motion prior based on the forecast horizon, preserving short-term kinematic continuity while allowing for significant reorientation when necessary.

D. Training Objective

The model is trained using a Smooth-L1 loss that assigns greater weight to later horizons to penalize long-term drift, balancing translation and rotation supervision.

3. Key Contributions

1. Problem Formulation: The paper formulates egocentric head motion prediction as a task conditioned on observed hand motion and inferred target context, explicitly coupling body motion dynamics with perceptual demands.
2. HAP Framework: A novel reasoning framework that bridges target intent and future head motion via:
   • A dynamic P-TAOG that encodes soft target probabilities and evolving directed occlusion relations.
   • A horizon-wise gating mechanism that balances learned motion with constant velocity priors.
3. Bottle Dataset: The introduction of Bottle, a new egocentric RGB-D dataset featuring object manipulation in cluttered scenes with changing target visibility. It includes coordinated head and hand motion data specifically designed to test prediction under frequent occlusion.

4. Experimental Results

The authors evaluated HAP on the public EgoPAT3Dv2 dataset and the new Bottle dataset, comparing against baselines including constant-velocity heuristics, recurrent models (RVAE, VRNN), attention-based models, and diffusion-based approaches.

• Performance: HAP achieved the lowest translation errors and competitive rotation errors on EgoPAT3Dv2. On the Bottle dataset, HAP outperformed all representative baselines (including Uni-Hand and diffusion models) across all four metrics (tADE, tFDE, rADE, rFDE), showing the largest margin in translation error.
• Ablation Studies:
  • P-TAOG & Gate: Both components were found essential. The horizon-wise gate improved performance across all variants, and P-TAOG provided significant gains when paired with the gate.
  • Graph Design: The directed graph with predictive occlusion features was critical for minimizing rotation errors, while current occlusion features aided translation accuracy.
  • Soft vs. Hard Targets: Using a soft target distribution (preserving uncertainty) yielded better results than hard assignment, indicating that uncertainty carries useful information for downstream prediction.
• Robustness: The framework remained stable when the localization backend was switched from Depth Anything 3 to ORB-SLAM3.

5. Significance and Claims

The paper claims that HAP demonstrates the value of hand-driven intention and dynamic occlusion reasoning for anticipating human head motion. By treating head motion as an active perception process rather than a passive consequence of hand movement, the framework successfully predicts 6-DoF trajectories that account for the need to reveal occluded targets.

The authors note that their evaluation is currently limited to tabletop datasets where head motion is represented by a rigidly mounted camera. They acknowledge that the quadratic edge scaling of P-TAOG with the number of objects may be computationally costly in highly cluttered scenes. Future work is proposed to evaluate HAP in larger, more diverse interaction settings and to integrate the framework with robotic grasping systems to investigate coordinated upper-body motion.