Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning

1. Problem Definition

Weakly-Supervised Video Scene Graph Generation (WS-VSGG) aims to parse video content into structured relational triplets $\langle \text{subject}, \text{predicate}, \text{object} \rangle$ without requiring dense bounding box annotations. Instead, it relies on sparse temporal supervision (e.g., unlocalized triplets annotated only on the middle frame of a video clip).

The Core Challenge:
Existing WS-VSGG methods typically adopt a two-stage pipeline:

1. Object Detection: Use off-the-shelf detectors (trained on static images) to generate object proposals for all frames.
2. Relation Prediction: Match these proposals to unlocalized triplets based on class identity to create pseudo-labels.

The Fundamental Discrepancy:
The authors identify a critical distribution shift between Fully-Supervised (FS) and Weakly-Supervised (WS) regimes:

• FS Detectors: Implicitly filter out non-interactive objects because they are trained on datasets where bounding boxes are only drawn for objects involved in relationships.
• WS Detectors: Off-the-shelf detectors indiscriminately detect all visible objects. This floods the relation prediction model with a massive number of non-interactive pairs (noise) that were never seen during training (where only interactive pairs were labeled).
• Consequence: The model is trained exclusively on "clean" interactive pairs but must infer on a test set dominated by noisy, non-interactive pairs, leading to a severe train-test distribution gap and poor performance.

2. Methodology

The authors propose a unified framework consisting of three key components to address the detection distribution shift and improve contextual reasoning.

A. Relation-Aware Matching (RAM)

• Problem: Standard class-level matching assigns all detections of the same class as positive pseudo-labels. If a frame contains multiple "cups," and the triplet is "person holding cup," standard matching might label a cup on the table as positive, creating false supervision.
• Solution: RAM leverages Vision-Language (VL) grounding models (specifically GroundingDINO) to resolve class-level ambiguity.
  • It constructs relation-aware text queries (e.g., "cup that the person is holding").
  • It uses cross-modal attention maps to localize the specific object instance that satisfies the relational context.
  • A Reliability Estimation mechanism checks the spatial concentration of the attention map. If reliable, it performs grounded matching; otherwise, it falls back to class-level matching.
• Outcome: Produces cleaner pseudo-labels by selecting only the interaction-consistent instance, reducing false positives in the training partition.

B. Pair Affinity Learning and Scoring (PALS)

• Problem: The relation prediction model needs to distinguish between interactive and non-interactive pairs, but standard pipelines discard unmatched pairs entirely, losing valuable negative signals.
• Solution:
  • Dual Embeddings: The model maintains two parallel embeddings for each pair: a Relation Embedding (for predicate classification) and a Pair Affinity Embedding (for estimating interaction likelihood).
  • Joint Training: Both matched (positive) and unmatched (negative) pairs are used to train the Pair Affinity head. This teaches the model to output a scalar score ($PA \in [0,1]$) indicating the likelihood of interaction, regardless of the specific predicate.
  • Inference Ranking: The final triplet score is modified to include the pair affinity score:
    $$\text{Score}(s, p, o) = \text{conf}_s \cdot \text{conf}_o \cdot \text{PC}_p \cdot \text{PA}_{(s,o)}$$
    This suppresses non-interactive pairs even if they have high detection confidence.

C. Pair Affinity Modulation (PAM)

• Problem: Even with better ranking, the internal contextual reasoning (via spatial/temporal attention layers) is corrupted by non-interactive pairs. In standard attention mechanisms, non-interactive pairs attend to each other, diluting the relational context.
• Solution: PAM acts as a plug-and-play module that gates the attention mechanism.
  • It constructs a pairwise affinity matrix $G_i = P_i P_i^\top$ from the pair affinity embeddings.
  • This matrix is used to modulate the attention logits via element-wise multiplication with a sigmoid function: $\text{Attn} = \text{Softmax}(\frac{QK^T}{\sqrt{d}} \odot \sigma(G))$.
  • Effect: This gates the information flow, allowing interactive pairs to attend to each other while suppressing the influence of non-interactive pairs on the contextual reasoning process.

3. Key Contributions

1. Identification of Distribution Shift: The paper highlights the fundamental mismatch between FS and WS detection pipelines, where WS models are overwhelmed by non-interactive objects.
2. Pair Affinity Learning (PALS): Introduces a learnable mechanism to explicitly model the boundary between interactive and non-interactive pairs, utilizing unmatched pairs as negative supervision rather than discarding them.
3. Relation-Aware Matching (RAM): Proposes a preprocessing step using VL grounding to resolve class-level ambiguity in pseudo-label generation, significantly improving label quality.
4. Pair Affinity Modulation (PAM): A novel attention gating mechanism that prevents non-interactive pairs from corrupting the contextual reasoning of the backbone network.
5. State-of-the-Art Performance: Demonstrates that explicitly modeling pairwise interactivity is crucial for WS-VSGG, narrowing the gap to fully-supervised performance significantly.

4. Experimental Results

• Dataset: Evaluated on Action Genome (AG), the standard benchmark for video scene graphs.
• Baselines: Compared against PLA (the standard weakly-supervised baseline) and TRKT, using STTran and DSG-DETR as backbones.
• Performance:
  • The proposed method achieved State-of-the-Art (SOTA) performance across all metrics.
  • Gains: Average improvement of +5.47 in Recall@10 (With Constraint) and +6.43 (No Constraint) over the best existing weakly-supervised baselines.
  • Gap Reduction: The method achieved 88.3% and 94.3% of the fully-supervised upper bound performance, significantly narrowing the gap between weak and full supervision.
• Ablation Studies:
  • RAM: Improved pseudo-label precision by ~80% (though slightly reduced recall), proving the value of cleaner supervision.
  • PALS: Provided the largest individual gain, confirming that filtering non-interactive pairs is the primary bottleneck.
  • PAM: Consistently improved performance, especially in scenarios with high ratios of non-interactive pairs.
• Efficiency: The added computational overhead is negligible (~1.27% increase in parameters, <0.05% increase in FLOPs).

5. Significance

This work fundamentally shifts the paradigm for weakly-supervised video understanding. Instead of treating the abundance of non-interactive objects as a nuisance to be ignored, the authors propose explicitly modeling the "non-interaction" signal. By learning to distinguish interactive from non-interactive pairs and modulating attention accordingly, the framework effectively adapts powerful fully-supervised architectures to the weakly-supervised regime. This approach is model-agnostic and can be applied to various backbones, offering a robust solution for scalable video scene graph generation with minimal annotation costs.