Technical Summary: AlignG – Learning Context-Conditioned Predicate Semantics via Prototype Feedback

1. Problem Statement

Scene Graph Generation (SGG) aims to represent images as structured graphs of objects and their semantic relationships. A fundamental challenge in this domain is modeling polysemous predicates, where the meaning of a label (e.g., "on," "riding") shifts depending on the specific visual context. For instance, a person standing on skis without motion is "standing on" them, whereas a person moving downhill is "riding" them, despite similar visual appearances.

Existing approaches attempt to address this by:

• Static Prototypes: Assigning a single fixed prototype per predicate (e.g., PE-Net).
• Multi-Prototype Decomposition: Splitting predicates into multiple static sub-prototypes (e.g., C-SGG).
• Retrieval/External Knowledge: Using external exemplars or Large Language Models (LLMs) for disambiguation.

However, these methods suffer from a critical limitation: predicate representations remain static. They cannot reorganize semantics to reflect image-specific evidence during inference. Fixed anchors fail to adapt to contextual nuances, leading to systematic confusions in ambiguous scenes where visual features alone are insufficient to distinguish between spatial contact and functional usage.

2. Methodology: AlignG

The authors propose AlignG, a prototype feedback learning framework that bridges global predicate semantics with image-specific relational context. Unlike prior works that treat prototypes as fixed anchors, AlignG treats them as context-conditioned variables that adapt dynamically based on relational cues within each image.

The framework operates via a two-stage refinement process:

Stage 1: Contextualizing Predicate Prototypes

Instead of using static prototypes, AlignG updates them incrementally using relational evidence from the current image.

• Mechanism: It aggregates relation embeddings for each prototype via compatibility-weighted cross-attention. This mechanism evaluates the compatibility between a global prototype and all relation candidates in the image, assigning higher weights to the most relevant ones.
• Update Rule: The resulting contextual signal is fed into a Gated Recurrent Unit (GRU) cell alongside the normalized static prototype. This produces an image-specific refined prototype $p^{(I)}_r$.
• Goal: This allows prototypes to evolve with local evidence while maintaining stability within the global semantic space, preventing abrupt semantic drift.

Stage 2: Recalibrating Relation Embeddings

Once prototypes are adapted, they are used to refine the relation embeddings.

• Mechanism: A reverse cross-attention mechanism propagates the adapted prototypes back to the relation candidates. This determines how much each adapted prototype contributes to refining a specific relation embedding.
• Recalibration: The raw relation embedding is fused with the prototype-informed feedback signal via a projection network.
• Goal: This single-step adjustment ensures relation representations are flexible to local variation yet consistent with global semantics.

Training Objectives

The optimization follows three key losses:

1. Classification Loss ($L_{cls}$): Standard cross-entropy for predicate prediction.
2. Regularization Loss ($L_{reg}$): Enforces diversity among prototypes and prevents redundancy (similarity penalty and diversity margin).
3. Alignment Loss ($L_{align}$): A contrastive loss that pulls recalibrated relation embeddings closer to their static global prototypes (not the adapted ones) and pushes them away from confusing negatives.
   • Crucial Design Choice: Anchoring the loss to static prototypes prevents trivial co-adaptation between relations and prototypes within a single image, ensuring the classifier remains grounded in image-agnostic semantic centers.

3. Key Contributions

1. Reformulation of Predicate Learning: The paper shifts the paradigm from static-prototype learning to image-conditioned adaptation, treating predicate semantics as dynamic variables responsive to relational evidence.
2. AlignG Framework: Introduction of a two-stage prototype feedback mechanism that:
   • Adapts prototypes to image-specific evidence.
   • Recalibrates relation features under these adapted prototypes.
   • Preserves global semantic coherence through static anchoring in the loss function.
3. Empirical Validation: Demonstration of state-of-the-art performance on standard benchmarks, supported by visualizations showing coherent, context-dependent reorganization of predicate semantics.

4. Experimental Results

The model was evaluated on VG-150 and GQA-200 under three settings: Predicate Classification (PredCls), Scene Graph Classification (SGCls), and Scene Graph Detection (SGDet).

• VG-150 Performance:

  • Under SGDet, AlignG achieved an F@100 of 23.8, a +1.4 improvement over the previous best (MCL).
  • It achieved the highest Mean Recall (mR@100) across all settings (42.6 in PredCls, 26.1 in SGCls, 19.7 in SGDet).
  • Compared to the PE-Net backbone, AlignG showed substantial gains in mR@100 (+8.8, +7.2, +5.2 respectively), confirming improvements stem from the feedback mechanism rather than concept expansion.

• GQA-200 Performance:

  • AlignG achieved an F@100 of 19.5 in SGDet, a +2.7 improvement over the baseline.
  • It demonstrated consistent improvements in mR@100 (+10.8 over PE-Net), effectively balancing global accuracy and fairness across long-tail distributions.

• Ablation & Analysis:

  • Component Analysis: Replacing simple concatenation with a GRU-based updater consistently improved performance, particularly in Mean Recall.
  • Confusion Analysis: AlignG resolved between 11.5% and 42.6% of errors made by PE-Net on hard-negative pairs (e.g., distinguishing "lying on" vs. "laying on").
  • Computational Overhead: The method adds minimal computational cost (+7.05G FLOPs, +0.03s/iter) while maintaining real-time inference speeds (>30 FPS).

5. Significance and Claims

The paper claims that semantic coherence and contextual flexibility can coexist within a unified structure. By integrating adaptability directly into predicate representations, AlignG demonstrates that static prototypes (providing stability) and image-specific adaptation (capturing scene evidence) can operate as complementary mechanisms.

The authors emphasize that their approach:

• Refines Ambiguous Predicates: It selectively merges or separates predicate semantics based on scene evidence, as visualized in per-image prototype similarity heatmaps.
• Avoids Semantic Drift: The use of static prototypes in the alignment loss anchors the model to global semantic centers, preventing the model from drifting toward local artifacts or noisy detections.
• Generalizability: While validated on image-based benchmarks, the authors suggest the principle of context-conditioned adaptation could extend to sequential settings like video understanding, where relational semantics shift across time.

The paper concludes with a modest impact statement, acknowledging that while the method advances SGG, it inherits annotation biases from training datasets. They recommend validation on target distributions and human oversight for safety-sensitive applications.