Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation

1. Problem Definition

The paper addresses Short-Term Object Interaction Anticipation (STA), a task defined by Grauman et al. [22] within the Ego4D dataset. The goal is to predict the future state of an egocentric video scene based on a short observation window. Specifically, given an input consisting of a high-resolution image frame ($I_T$) and a sequence of preceding low-resolution video frames ($V_{T-t:T}$), the model must simultaneously predict:

• Next-Active Objects: The bounding box ($b_m$) of the object the user is about to interact with.
• Semantic Categories: The noun (object category, $n_m$) and verb (interaction mode, $v_m$).
• Temporal Dynamics: The time-to-contact ($\delta_m$), indicating when the interaction will occur.
• Confidence: A score ($s_m$) for the prediction.

This task is critical for wearable assistants and human-robot interaction, requiring the system to understand not just what is in the scene, but what will happen next based on human intent and environmental context.

2. Methodology

The authors propose a two-fold approach: a novel transformer-based architecture (STAformer and STAformer++) and a grounding mechanism using Environment Affordances and Interaction Hotspots.

A. Architecture: STAformer and STAformer++

Both models utilize an image-video pair input but differ in their detection heads and backbone refinements.

1. Feature Extraction:

   • Image: Uses pre-trained transformers (DINOv2 for STAformer, Swin-T for STAformer++) to extract high-resolution 2D features.
   • Video: Uses video transformers (TimeSformer for STAformer, EgoVideo for STAformer++) to extract spatio-temporal 3D features from the preceding frame sequence.

2. Novel Attention Components:

   • Frame-Guided Temporal Pooling: A mechanism that maps 3D video tokens to the 2D spatial reference system of the last observed frame. It uses residual cross-attention where the last frame acts as the query, ensuring temporal features are spatially aligned with the current image.
   • Dual Image-Video Attention: A dual cross-attention module that enriches image tokens with scene dynamics (from video) and video tokens with fine-grained visual details (from image). This allows the model to capture both static object details and motion context.

3. Detection Heads:

   • STAformer: Uses a Faster R-CNN based head (adapted from StillFast) with a Region Proposal Network (RPN) and RoI Align.
   • STAformer++: Replaces the CNN-based head with a DETR (Detection Transformer) head (specifically DINO-DETR). This end-to-end approach eliminates hand-crafted anchors and NMS, utilizing object queries and deformable attention for improved detection accuracy and convergence.

B. Grounding Predictions: Affordances and Hotspots

To improve generalization and reduce false positives, the authors integrate external knowledge about human behavior.

1. Environment Affordances:

   • Concept: Based on Gibson's theory, affordances represent potential actions an environment offers. The authors build a persistent memory (database) of "affordance zones" by clustering training videos based on visual and textual similarity.
   • Implementation:
     • Fixed (Inference-time): Matches the input video to the top-$K$ similar zones in the database using cosine similarity. The noun/verb probabilities from these zones are used to re-weight the model's predictions via late fusion.
     • Learned (Training-time): An attention mechanism learns to dynamically associate the input video with relevant zones in the database during training. This allows the model to adaptively refine its predictions based on a flexible memory of past human behaviors in similar environments.

2. Interaction Hotspots:

   • A module predicts a probability map of where interactions are likely to occur by observing hand and object trajectories.
   • Refinement: The confidence scores of the STA predictions are re-weighted based on the predicted hotspot probability at the center of the bounding box. This suppresses false positives in areas where interactions are unlikely to happen.

3. Key Contributions

1. STAformer & STAformer++ Architectures: Two novel attention-based architectures that effectively fuse image and video modalities. STAformer++ achieves state-of-the-art performance by integrating a DETR-based detection head and advanced feature extractors (Swin-T, EgoVideo).
2. Affordance Integration: The first unified framework to integrate environment affordances into short-term anticipation. They propose both a fixed late-fusion strategy and a novel learned attention mechanism that grounds predictions in a persistent memory of human behavior.
3. Interaction Hotspot Refinement: A method to spatially re-weight prediction confidence based on predicted hand trajectories and interaction zones.
4. New Benchmark & Resources: The authors curated and released a new set of STA annotations for the EPIC-Kitchens dataset, along with code, pre-extracted affordances, and annotations for Ego4D and EPIC-Kitchens.

4. Results

The models were evaluated on the Ego4D (v1 and v2) and the new EPIC-Kitchens STA benchmarks using Top-5 Mean Average Precision (mAP).

• Ego4D v1 Validation:
  • STAformer++ with learned affordances achieved 4.66 All mAP, a +23.6% relative gain over the previous conference version (STAformer) and significantly outperforming baselines like StillFast (2.48) and GANO (3.99).
  • Significant improvements were seen in semantic metrics (Noun mAP: +36.3%, Noun+Verb mAP: +24.5%).
• Ego4D v2 Validation:
  • STAformer++ achieved 6.26 All mAP, a +10.4% relative gain over the second-best method.
• EPIC-Kitchens:
  • The model demonstrated strong generalization, achieving 8.67 All mAP, a massive +31.5% improvement over the baseline StillFast (3.28).
• Ablation Studies:
  • Replacing Faster R-CNN with DETR (STAformer++) provided the largest single boost in detection quality.
  • The "Learned" affordance approach outperformed the "Fixed" approach, proving the value of adaptive memory over static similarity matching.
  • Frame-guided temporal pooling and dual cross-attention were critical for aligning spatio-temporal features.

5. Significance

This work represents a significant leap in egocentric video understanding by moving beyond simple sequence modeling to context-aware anticipation.

• Architectural Advancement: It demonstrates that transformer-based detection heads (DETR) are superior to CNN-based heads for complex anticipation tasks involving both spatial and temporal reasoning.
• Behavioral Grounding: By integrating affordances, the model learns to anticipate actions based on where and how humans typically interact with objects in specific environments, rather than just learning visual patterns. This improves robustness in long-tail scenarios.
• Community Impact: The release of curated EPIC-Kitchens STA labels and pre-computed affordance databases provides a valuable resource for future research in human-robot interaction and assistive AI.

In summary, the paper establishes a new state-of-the-art for Short-Term Object Interaction Anticipation by combining modern transformer architectures with psychologically grounded models of human-environment interaction.