Frequency-guided Multi-level Reasoning for Scene Graph Generation in Video

1. Problem Statement

Video Scene Graph Generation (VSGG) aims to translate video content into structured semantic representations (objects as nodes, relationships as edges) to facilitate high-level understanding. However, existing methods face significant challenges:

• Long-Tail Distribution: Predicate relationships in real-world videos follow a severe long-tail distribution. Frequent relationships (e.g., "holding") dominate the dataset, while rare relationships (e.g., "carrying") are underrepresented.
• Gradient Conflicts & Head Bias: In traditional models, a shared backbone network processes all relationship types. This leads to gradient conflicts, where the strong gradients from frequent ("head") classes suppress the learning of rare ("tail") classes, causing tail relations to underfit.
• Temporal Incoherence & Uncertainty: Processing frames independently causes inconsistencies, and real-world factors like occlusion and motion blur introduce reasoning uncertainty, which standard classifiers struggle to handle.

2. Methodology: FReMuRe

The authors propose the Frequency-guided Relational Multi-level Reasoning (FReMuRe) model. The core philosophy is to decouple the learning process based on predicate frequency to resolve gradient conflicts and enhance tail-class representation.

A. Frequency-Guided Mechanism

• Frequency Prior: The model calculates the statistical frequency ($f_k$) of each relationship type in the dataset.
• Frequency-Aware Gating: A learnable gating mechanism uses the inverse of the frequency ($\log(1/f)$) to scale input features.
  • Tail Enhancement: For rare relationships, the gate value increases, amplifying their features.
  • Stabilization: It employs a parameter-free LayerNorm ($LN$) to stabilize the distribution of tail features.
  • Formula: $x' = g(f) \odot LN(x) + (1 - g(f)) \odot x$, where $g(f)$ is a sigmoid function based on frequency. This acts as a structural correction rather than just a loss reweighting.

B. Dual-Branch Predicate Embedding Generator (DPEG)

To physically separate the learning of common and rare relationships:

• Architecture: The model splits into two parallel branches:
  1. Local Branch (Head): Standard encoding for high-frequency relationships.
  2. Tail Branch: A frequency-aware branch specifically designed to model low-frequency relationships.
• Decoupling: By dedicating independent parameters to each branch, the model prevents the gradients of head classes from interfering with tail classes, ensuring a dynamic balance.
• Global Context: A global decoder incorporates cross-frame information (temporal consistency) using positional encoding and sliding window aggregation to refine predictions across time.

C. Specialized Classification Heads

The paper introduces two interchangeable heads to handle uncertainty and intra-class diversity:

1. Bayesian Head: Models predictions as a distribution (Mean $\mu$ and Variance $\sigma^2$). It uses Monte Carlo sampling during training to capture both aleatoric uncertainty (data noise) and epistemic uncertainty (model ignorance), improving robustness for rare classes.
2. GMM-Plus Head: An extension of the Gaussian Mixture Model where each class is represented by $K$ Gaussian components.
   • Frequency-Aware Regularization: It enforces a lower bound on variance to prevent Gaussian components from degenerating into Dirac distributions (overfitting) when tail samples are scarce. This allows the model to capture diverse visual modes for rare relationships.

3. Key Contributions

1. Frequency-Guided Tail-Class Debiasing: Introduced a structural gating mechanism that uses frequency priors to adaptively recalibrate features, significantly improving the recall of low-frequency relationships.
2. Decoupled Dual-Branch Network: Proposed a Transformer-based architecture that physically separates the learning of head and tail predicates, effectively resolving gradient conflicts inherent in shared backbones.
3. Replaceable Classification Heads: Developed two advanced heads (Bayesian and GMM-Plus) that enhance robustness through uncertainty estimation and multi-prototype modeling, respectively.

4. Experimental Results

Experiments were conducted on the Action Genome (AG) dataset, a benchmark for dynamic scene graphs.

• Metrics: Evaluated using Recall@K (R@K) and Mean Recall@K (mR@K), with the latter being crucial for long-tail performance.
• Performance:
  • PREDCLS (Predicate Classification): FReMuRe+GMM-Plus achieved a state-of-the-art (SOTA) mR@50 of 43.1, surpassing previous methods like TEMPURA and STTran-TPI.
  • SGCLS (Scene Graph Classification): Set new SOTA across all metrics, reaching mR@50 of 30.6.
  • SGDET (Scene Graph Detection): Achieved the best mR@50 of 22.5, demonstrating superior ability to detect diverse relationships end-to-end.
• Ablation Studies: Removing the decoupling strategy caused the most severe performance drop (SGDET mR@50 fell from 22.5 to 17.3), confirming that separating learning pathways is the most critical factor. Removing frequency guidance or specialized heads also resulted in significant degradation.
• Qualitative Analysis: Visual comparisons showed FReMuRe correctly identifying rare interactions (e.g., "person standing on floor") that baseline models (like TEMPURA) misclassified as common relationships (e.g., "in front of").

5. Significance

This paper addresses a fundamental limitation in video understanding: the inability of current models to learn rare but semantically important relationships due to data imbalance. By shifting from a unified feature space to a frequency-guided, decoupled architecture, FReMuRe provides a robust framework for:

• Balanced Learning: Eliminating the bias toward frequent classes without sacrificing overall accuracy.
• Robust Reasoning: Handling uncertainty and noise inherent in video data through probabilistic modeling.
• Generalizability: Offering a modular design (swappable heads) that can be adapted to various long-tail distribution challenges in computer vision.

The source code is publicly available, facilitating further research into long-tail video understanding.