Hand Trajectory Fusion for Egocentric Natural Language Query Grounding

Technical Summary: Hand Trajectory Fusion for Egocentric Natural Language Query Grounding

Problem Definition
Egocentric Natural Language Query (NLQ) grounding aims to localize the temporal interval $[t_s, t_e]$ within a long first-person video that corresponds to a free-form text query. While state-of-the-art systems (e.g., GroundNLQ) effectively fuse video appearance with text semantics using large pretrained encoders, they often ignore hand motion. This is a significant oversight given that approximately 41% of queries in the Ego4D NLQ dataset are answered during moments of hand–object manipulation or their immediate outcomes. Existing auxiliary modalities, such as gaze or object detection, have been explored, but hand trajectory as a distinct modality for NLQ grounding remains unexplored despite its centrality to first-person activity.

Methodology
The proposed approach introduces a Hand-Trajectory Encoder and a Trajectory Fusion module to integrate sparse hand skeleton data with video-text representations. The architecture consists of five main components:

1. Encoders:

   • Video Encoder: Uses frozen, pretrained backbones (InternVideo and EgoVLP) to embed egocentric clips into video tokens ($E_v$).
   • Text Encoder: Uses a frozen CLIP model to embed natural language queries into text tokens ($E_t$).
   • Trajectory Encoder: A trainable module that processes a temporal sequence of hand skeletons (extracted via MediaPipe) into dense, video-aligned kinematic features ($E_h$). It employs a spatio-temporal transformer factorized into two stages:
     • Spatial Aggregation: Uses cross-attention to pool 21 anatomical landmarks per hand into a single frame-level descriptor, explicitly masking undetected landmarks.
     • Temporal Modeling: Applies self-attention to model the evolution of these descriptors across frames, capturing manipulation dynamics (approach, contact, release). Undetected frames are masked to prevent information leakage.

2. Trajectory Fusion:

   • This module integrates $E_h$ and $E_t$ into the video tokens $E_v$ using a cross-attention with adaptive gating strategy.
   • Two independent cross-attention mechanisms allow video tokens to query the trajectory and text streams, producing enriched representations ($E_{vh}$ and $E_{vt}$).
   • Adaptive Gating: Instead of uniform summation, the contributions of the trajectory and text branches are scaled by learned scalar gates ($g_h$ and $g_t$). These gates are computed via lightweight MLPs applied to the temporal averages of the respective cross-attention outputs. This allows the network to selectively attenuate the trajectory signal when hand detection is sparse or irrelevant.
   • A self-attention refinement block further processes the merged representation to yield the final multimodal output ($E_o$).

3. Prediction:

   • A Temporal Segment Prediction head predicts the answer span $[t_s, t_e]$ from the fused representation $E_o$.

Key Contributions

• Novel Modality: The work is the first to study hand trajectory as an auxiliary modality for egocentric NLQ grounding, addressing a gap where roughly 41% of queries rely on manual activity.
• Robust Trajectory Encoding: The design explicitly handles the sparsity of hand detection (hands are visible in only ~41% of Ego4D frames on average) through masking mechanisms in both spatial and temporal transformer stages.
• Adaptive Fusion: The introduction of learned, content-dependent gating mechanisms allows the model to dynamically weigh the hand trajectory signal against video and text features, preventing noise from undetected frames from degrading performance.

Experimental Results
The system was evaluated on the Ego4D NLQ v2 validation split (4,552 query-clip pairs) using the $R_m@IoU=n$ metric.

• Category-Specific Gains: The most significant improvements were observed in categories directly related to manipulation:
  • Hand–Object Interaction (HOI): +2.54 improvement in $R1@IoU=0.3$.
  • Quantity/State: +4.32 improvement in $R1@IoU=0.3$.
  • Within HOI, gains were concentrated in action templates (e.g., "What did I put in X?"), confirming that trajectory cues help localize the specific moment of action.
• Overall Performance: The proposed model achieved a +0.77 gain in overall $R1@IoU=0.3$ and a +1.39 gain in $R1@IoU=0.5$. The larger gain at the stricter IoU threshold suggests that hand kinematics not only help retrieve the correct temporal region but also sharpen the localization precision.
• Efficiency: The Trajectory Encoder adds only 195K parameters (0.6% of the full model) and is trained from scratch while keeping the heavy video and text backbones frozen.

Significance and Limitations
The paper claims that hand kinematics provide a lightweight yet informative signal that complements visual appearance, particularly for queries involving object manipulation. The design demonstrates that auxiliary modalities can be effectively integrated without modifying pretrained backbones.

The authors modestly acknowledge a primary limitation: detection sparsity. Since hands are visible in only 41% of frames, the trajectory branch's contribution is inherently capped. The paper posits that future improvements in egocentric hand detection would directly translate to stronger grounding performance. Additionally, the Trajectory Fusion module is described as modality-agnostic, suggesting it could naturally extend to other signals like gaze or larger-scale training scenarios.