Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipation

1. Problem Statement

Video-based Human-Object Interaction (HOI) understanding requires two capabilities: detecting ongoing interactions and anticipating their future evolution. Current approaches suffer from two fundamental limitations:

• Decoupled Architecture: Existing methods typically follow a multi-stage pipeline: first detecting/tracking humans and objects, then constructing pairs, and finally classifying or forecasting interactions. This treats anticipation as a downstream prediction task over externally constructed candidates rather than a structured evolution of the pair representation. Consequently, the model fails to learn how pair identity and state persist or change over time, leading to weak temporal constraints.
• Evaluation Flaws: Standard benchmarks (e.g., VidHOI, Action Genome) rely on sparse keyframe annotations. This creates temporal discontinuities where "future" labels are separated from observed clips by irregular gaps. As a result, reported anticipation performance often reflects annotation sparsity or clip-construction artifacts rather than genuine reasoning over continuous dynamics.

2. Methodology: HOI-DA

The authors propose HOI-DA, a unified, pair-centric framework that jointly performs subject-object localization, present HOI detection, and multi-horizon anticipation.

Core Philosophy

Instead of predicting future interactions independently, HOI-DA models future states as structured residual transitions from the current pair state. Anticipation is not an add-on but a structural constraint on how pair-level interactions are represented over time.

Architecture Components

1. Unified Pair-Centric Decoder:

   • Pair-Slot Instance Decoder: Uses learnable query tensors ($Q_s, Q_o$) to maintain persistent human-object pair slots across time. Unlike tracking-based methods, pair identity is preserved internally within the model via aligned slots, eliminating the need for external association.
   • Present Interaction Decoder: Grounds the shared pair representation in the observed clip to predict current interactions.
   • Temporal Summary Module: Instead of extrapolating from the last frame, this module uses learnable horizon anchors to attend over the entire observed interaction history, creating a history-conditioned summary for future prediction.
   • Horizon-Conditioned Anticipation Decoder: Predicts future verb labels at specific time horizons ($h$) by modeling the residual ($R^{(h)}$) between the future state and the observation-boundary state. This forces the model to focus on change rather than re-describing the current state.

2. Language-Guided Semantic Branch:

   • To handle long-tail verb distributions, a lightweight branch uses a pretrained text encoder (RoBERTa) to inject vocabulary-level semantic structure into object and verb predictions. This acts as a semantic prior that remains causally valid for future prediction (independent of unobserved frames).

3. Dual Orthogonality Regularization:

   • To prevent future predictions from collapsing into the current state or becoming redundant across horizons, the authors introduce two constraints:
     • Task Orthogonality: Separates the representation of the present state from the future change (residual).
     • Horizon Orthogonality: Ensures that predictions for different time horizons are distinct and non-redundant.

4. Training Objective:

   • Uses shared bipartite matching for both detection and anticipation tasks.
   • Employs a geometric schedule to weight losses, emphasizing near-term forecasting while gradually ramping up anticipation terms to stabilize joint optimization.

3. Key Contributions

A. HOI-DA Framework

A novel one-stage architecture that unifies detection and anticipation. By treating pair persistence as an architectural primitive and modeling future dynamics as residuals, it achieves temporally consistent reasoning.

B. Dual Orthogonality Regularization

A novel loss mechanism that structurally separates present grounding, future change, and horizon-specific dynamics, preventing representation collapse and redundancy.

C. DETAnt-HOI Benchmark

A temporally corrected benchmark derived from VidHOI and Action Genome.

• Problem Addressed: The original datasets contain large temporal gaps between keyframes, making "future" labels unreliable.
• Solution: The authors supplement short non-interactive gaps with bounding-box annotations (from VidOR) to maintain instance continuity and split videos at long inactive intervals. This ensures future labels correspond to actual continuous dynamics rather than annotation artifacts.

4. Experimental Results

The model was evaluated on the corrected DETAnt-HOI benchmark (VidHOI and Action Genome components).

• Performance: HOI-DA outperforms state-of-the-art two-stage baselines (e.g., ST-HOI, STTran, Gaze-Tran) across all metrics (mAP and Recall@k).
  • On VidHOI, HOI-DA improves detection mAP by +5.87 (Full) and anticipation mAP by +8.59 at horizon $h=7$ compared to the strongest baseline (Gaze-Tran).
  • The performance gap widens as the forecasting horizon increases, proving the model's superior ability to handle long-term temporal uncertainty.
• Ablation Studies:
  • Removing the Temporal Summary Module significantly degrades long-horizon performance, confirming the necessity of history-conditioning.
  • Removing Dual Orthogonality hurts anticipation more than detection, validating its role in separating future dynamics.
  • A naive "Unified One-Decoder" (without structural constraints) performs worse than separate baselines, proving that simple parameter sharing is insufficient; structured coupling is required.
• Qualitative Analysis: The model successfully maintains pair identity through occlusion and camera motion (where two-stage pipelines fail) and captures transient relations (e.g., a "watch" interaction appearing briefly) that boundary-based extrapolation misses.

5. Significance

This work fundamentally shifts the paradigm of video HOI understanding:

1. From Pipeline to Unified Representation: It demonstrates that anticipation is most effective when treated as a structural constraint on pair-level representation learning, rather than a downstream task.
2. Evaluation Integrity: By introducing DETAnt-HOI, the paper highlights that previous evaluations were flawed by temporal discontinuities, urging the community to adopt more faithful evaluation protocols.
3. Robustness: The approach provides a robust solution for applications requiring long-term reasoning, such as collaborative robotics and proactive safety monitoring, where understanding the evolution of interaction is as critical as detecting the current state.

In summary, HOI-DA achieves state-of-the-art results by unifying detection and anticipation through a residual, pair-centric architecture, supported by a rigorously corrected benchmark that ensures evaluation reflects true temporal reasoning capabilities.