Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation

1. Problem Definition

The paper addresses Temporal Panoptic Scene Graph Generation (TPSGG), a task that aims to generate dynamic scene graphs from video inputs (including natural videos, RGB-D, and 4D point clouds).

• Goal: To represent entities as nodes and their spatio-temporal relationships as edges, capturing both static (e.g., "standing on") and dynamic (e.g., "kicking," "opening") interactions.
• Limitations of Existing Methods: Current state-of-the-art approaches typically follow a two-stage pipeline:
  1. Segmentation & Tracking: Generate "mask tubes" (sequences of segmentation masks tracked across time).
  2. Relation Prediction: Encode these tubes using global pooling (e.g., average pooling) followed by an MLP to classify relations.
• The Core Issue: Global pooling operations flatten the temporal dimension, effectively discarding motion patterns and frame ordering. This leads to poor performance on dynamic relations, as the model relies heavily on static visual semantics rather than the evolution of movement. Existing contrastive learning methods for videos often focus on frame-level semantics or whole clips, failing to isolate motion specific to entity mask tubes.

2. Methodology

The authors propose a novel Motion-aware Contrastive Learning Framework designed to force the model to learn representations sensitive to motion trajectories rather than just static appearance.

A. Framework Overview

The framework integrates with a standard TPSGG backbone (Panoptic Segmentation + Relation Classification) but introduces a specific contrastive learning objective during training to refine the entity mask tube representations ($H_i$).

B. Contrastive Learning Strategies

The objective is to pull together representations of similar motion patterns and push apart those with distinct patterns.

1. Positive Sampling:
   • Selects mask tubes from different videos that share the same subject-relation-object (S-R-O) triplet category (e.g., "person kicking ball" from Video A and "person kicking ball" from Video B).
   • Rationale: Since visual semantics differ between videos, the model is forced to rely on the shared motion trajectory to associate the anchor and positive samples.
2. Negative Sampling (Two Strategies):
   • Shuffle-based: Takes the anchor tube and temporally shuffles the frames ($\pi(H_a)$).
     • Challenge: Static relations (e.g., "on") have little motion, making shuffled versions identical to the original.
     • Solution: The authors introduce a Strong-Motion Selection Strategy using optical flow edges to filter and select only tubes with significant motion amplitude for shuffling.
   • Triplet-based: Selects negative tubes from the same video but with different S-R-O triplets.
     • Strategy: Uses a multi-nomial distribution to preferentially sample triplets that share visual categories (subject/object) with the anchor but have different relations. This creates "hard negatives" that force the model to distinguish relations based on motion rather than visual similarity.

C. Optimal Transport for Relation Quantification

A critical challenge is measuring similarity between two mask tubes of potentially different lengths (different event durations). Standard pooling fails here.

• Approach: The authors treat two mask tubes as discrete probability distributions ($\mu$ and $\nu$).
• Metric: They utilize Optimal Transport (OT) distance (specifically using the Sinkhorn algorithm) to find the minimum cost to transport one distribution to the other.
• Benefit: This effectively synchronizes the motion states of two triplets, preserving the temporal evolution and trajectory of the masks without requiring them to be of equal length. The similarity score is derived as $sim = \alpha - d_{OT}$.

3. Key Contributions

1. Motion-Aware Contrastive Framework: A novel learning paradigm that explicitly encourages the model to learn representations based on motion patterns by aligning similar S-R-O triplets across different videos and separating them from temporally shuffled or visually similar but semantically different triplets.
2. Optimal Transport for Tube Alignment: The introduction of Optimal Transport distance to quantify relationships between mask tubes of varying lengths, overcoming the limitations of temporal pooling in capturing motion dynamics.
3. Strong-Motion Selection: A heuristic strategy using optical flow to filter out static relations during negative sampling, ensuring the contrastive objective remains effective.
4. State-of-the-Art Performance: Demonstrated significant improvements over existing methods on both natural video and 4D datasets, particularly in recognizing dynamic relations.

4. Experimental Results

The method was evaluated on OpenPVSG (natural videos) and PSG4D (4D RGB-D and Point Cloud videos).

• Datasets:
  • OpenPVSG: 400 videos (ViDOR, Epic-Kitchens, Ego4D).
  • PSG4D: Includes PSG4D-GTA (third-person) and PSG4D-HOI (egocentric).
• Metrics: Recall at K (R@K) and mean Recall at K (mR@K).
• Key Findings:
  • Overall Performance: The proposed method ("Ours") outperformed strong baselines (IPS+T-Convolution, IPS+T-Transformer, PSG4DFormer) across all metrics.
  • Dynamic Relations: As illustrated in Figure 1, the method significantly improved performance on dynamic predicates (e.g., "kicking," "running on") compared to baselines which favored static relations (e.g., "standing on").
  • 4D Data: Achieved substantial gains on both Point Cloud and RGB-D inputs, showing generalizability beyond standard 2D videos.
  • Ablation Studies:
    • Removing either the shuffle-based or triplet-based contrastive loss degraded performance, with triplet-based being slightly more fundamental.
    • Replacing Optimal Transport with standard Pooling + Cosine/L2 distance resulted in lower accuracy, confirming the necessity of preserving temporal structure.
    • The threshold $\gamma$ for selecting strong-motion tubes showed an optimal point; too low includes static noise, too high reduces data volume.

5. Significance

This work bridges the gap between panoptic segmentation and temporal reasoning. By shifting the focus from static visual features to motion-aware representations, the paper solves a critical bottleneck in scene graph generation: the inability to accurately model dynamic interactions.

• Technical Impact: It demonstrates that contrastive learning, when carefully designed with domain-specific constraints (mask tubes, optimal transport), can effectively extract temporal dynamics that standard pooling layers miss.
• Application: The improved ability to understand dynamic relationships is crucial for autonomous agents, robotics, and intelligent surveillance systems that must interact with or predict human actions in real-time.