💻 computer science

MOSA: Motion-Guided Semantic Alignment for Dynamic Scene Graph Generation

This paper proposes MoSA, a motion-guided semantic alignment framework that enhances Dynamic Scene Graph Generation by integrating object-pair motion attributes with spatial features, aligning visual representations with text embeddings via cross-modal matching, and employing a category-weighted loss to improve the modeling of tail relationships.

Original authors: Xuejiao Wang, Bohao Zhang, Changbo Wang, Gaoqi He

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuejiao Wang, Bohao Zhang, Changbo Wang, Gaoqi He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a busy movie scene. A person is sitting at a table, holding a sandwich, and looking at a dog.

Old computer programs trying to understand this scene are like a robot with bad eyesight and no sense of motion. They see the person and the sandwich, but they might get confused. Is the person holding the sandwich, or just touching it? Is the dog looking at the person, or just near them? Because the robot only looks at static snapshots (like a photo), it misses the subtle clues: the way the hand moves toward the mouth, or how the dog's head turns.

This paper introduces a new system called MoSA (Motion-Guided Semantic Alignment). Think of MoSA as a super-smart film critic who doesn't just look at the actors, but also watches how they move and listens to what the script says.

Here is how MoSA works, broken down into three simple steps:

1. The "Motion Detective" (MFE)

The Problem: Old systems treat a video like a stack of still photos. They miss the action.
The MoSA Solution: MoSA has a special "Motion Detective" module. Instead of just asking "What is that?", it asks:

"How fast are they moving?"
"Are they getting closer or farther apart?"
"Are they moving in the same direction?"

The Analogy: Imagine you are trying to guess if two people are about to hug or just walk past each other. If you only look at one frame, they might look close. But if you watch them move, you see one person leaning in (hug) while the other leans away (walk past). MoSA calculates these "motion clues" (speed, direction, distance) to figure out the true relationship.

2. The "Motion-Enhanced Translator" (MIM)

The Problem: Even if the robot sees the motion, it might not know how to mix that with the visual picture.
The MoSA Solution: This module is like a translator that combines the "Motion Detective's" notes with the "Visual Camera's" photos. It fuses them together so the computer understands that "moving closer + holding an object" = "eating," not just "holding."

The Analogy: It's like watching a dance. If you only see the dancers' positions (spatial), you might think they are just standing there. But if you combine their positions with the rhythm of their steps (motion), you realize they are performing a specific dance move. MoSA combines the "where" with the "how."

3. The "Script Reader" (ASM)

The Problem: Sometimes, two actions look very similar visually. For example, "drinking from a cup" and "touching a cup" might look almost the same in a blurry video.
The MoSA Solution: This module brings in language knowledge. It compares what it sees in the video against a library of text descriptions (like a script). It asks, "Does this visual pattern match the text description of 'drinking' better than 'touching'?"

The Analogy: Imagine you are trying to identify a song by humming a few notes. It's hard. But if you have a list of lyrics (the text) and you match your humming to the words, it becomes easy. MoSA matches the video action to the "lyrics" (text descriptions) to make sure it picks the right word.

Why is this a big deal? (The "Long-Tail" Problem)

In the world of data, some things happen all the time (like "person standing"), but some rare things happen very rarely (like "person juggling"). These rare things are called the "Long Tail."

Old computers are lazy; they guess the common things because they are easy. They often ignore the rare, interesting actions.

MoSA's Secret Weapon: It uses a special scoring system that forces the computer to pay extra attention to those rare, difficult relationships. It's like a teacher who gives extra credit for solving the hardest problems, ensuring the computer learns the rare stuff too.

The Result

When tested on a huge dataset of video clips (Action Genome), MoSA proved to be the best at:

Spotting the details: Distinguishing between "touching" and "drinking."
Understanding the story: Knowing that a person moving a sandwich to their mouth is "eating," not just "holding."
Finding the rare gems: Accurately identifying the weird, rare actions that other systems miss.

In short: MoSA is a video understanding system that doesn't just look at what is happening, but watches how it moves and checks the script to make sure it gets the story right, even for the tricky parts.

1. Problem Statement

Dynamic Scene Graph Generation (DSGG) aims to structurally model objects and their time-varying interactions in video sequences to enable high-level semantic understanding. Despite advancements, existing methods face three critical challenges:

Fine-grained Relationship Modeling: Current approaches often rely on static visual features and coarse temporal context, failing to distinguish between semantically similar actions that differ in motion patterns (e.g., distinguishing "touching" from "drinking from").
Semantic Representation Utilization: Many methods ignore textual prior knowledge, relying solely on visual data, which limits their ability to resolve ambiguities between relationships that look similar visually but have different semantic meanings.
Long-Tail Distribution: DSGG datasets suffer from severe class imbalance. Models are biased toward high-frequency relationships, leading to poor recognition of rare but meaningful "tail" relationships.

2. Methodology: The MoSA Framework

The authors propose MoSA (Motion-guided Semantic Alignment), a framework designed to explicitly model motion attributes and align them with semantic priors. The architecture consists of four core components:

A. Motion Feature Extractor (MFE)

To capture dynamic interactions, the MFE explicitly computes four motion attributes for every object pair $(o_i, o_j)$ based on bounding box trajectories:

Relative Distance ( $d_t$ ): The Euclidean distance between object centers, reflecting spatial interaction trends.
Approach Velocity ( $v_t$ ): The rate of change in distance, indicating if objects are moving closer or further apart.
Motion Persistence (Mean IoU): The average Intersection-over-Union over a sliding window, evaluating the stability or continuity of the relationship.
Directional Consistency ( $CosSim_t$ ): The cosine similarity of movement vectors, reflecting the coordination of object movements.
These attributes are concatenated and processed through an MLP to generate high-level motion features ( $F_{motion}$ ).

B. Motion-guided Interaction Module (MIM)

The MIM fuses the extracted motion features with spatial relationship features using an attention mechanism.

Mechanism: Motion features act as dynamic guidance, while spatial features (derived from a Spatial Encoder using CLIP visual embeddings and object categories) serve as context.
Output: This generates "motion-aware" relationship representations ( $F_{mim}$ ) that capture fine-grained actions difficult to distinguish by appearance alone. These are combined with original spatial features via residual connections to form joint features ( $F_{joint}$ ).

C. Temporal Decoder

To model the evolution of relationships over time, the joint features for each object pair are organized into trajectories and fed into a Temporal Transformer Decoder. This module captures long-term dependencies and complex temporal dynamics across the video sequence.

D. Action Semantic Matching (ASM)

To address semantic ambiguity and leverage prior knowledge, the ASM module aligns visual relationship features with text embeddings.

Process: Text descriptions for all possible relationship categories (e.g., "a person eating a sandwich") are encoded using a CLIP Text Encoder.
Alignment: The model computes the dot-product similarity between the visual relationship features and the text embeddings. This cross-modal matching provides explicit semantic guidance, helping the model distinguish between visually similar but semantically distinct relationships.

E. Loss Function & Long-Tail Strategy

To mitigate the long-tail bias, the authors introduce a Category-Weighted Loss.

The loss function assigns higher weights to rare categories based on their frequency ( $n_r$ ).
The weight $\alpha_r$ is calculated as inversely proportional to the logarithm of the category frequency, ensuring the model pays more attention to under-represented relationships during training.

3. Key Contributions

MoSA Architecture: A novel framework that explicitly models multidimensional motion attributes (distance, velocity, persistence, direction) to enhance the modeling of dynamic object interactions.
Motion-Guided Interaction (MIM): A mechanism that fuses motion and spatial features via attention, enabling the distinction of fine-grained actions that static features miss.
Action Semantic Matching (ASM): A cross-modal alignment mechanism that integrates textual priors to resolve semantic ambiguities and improve the discrimination of similar relationships.
Long-Tail Optimization: A category-weighted loss strategy that significantly improves the generalization and recognition accuracy of rare (tail) relationships.

4. Experimental Results

The method was evaluated on the Action Genome (AG) dataset across three standard tasks: Predicate Classification (PREDCLS), Scene Graph Classification (SGCLS), and Scene Graph Detection (SGDET).

State-of-the-Art Performance: MoSA outperformed existing SOTA methods (including TD2-Net, STTran, and TRACE) in both "With Constraint" and "No Constraint" settings.
- PREDCLS: Achieved 70.6% R@10 and 73.5% R@50, surpassing the previous best (TD2-Net).
- SGDET: Achieved 38.3% R@50, demonstrating strong detection capabilities.
Long-Tail Performance: MoSA showed significant improvements in mean Recall (mR@K), which is the metric for long-tail performance.
- In the SGDET task, MoSA improved mR@50 by 2.9% and 3.0% over the second-best methods under different settings.
Ablation Studies: Removing the MFE module caused the largest performance drop, confirming that explicit motion modeling is the most critical component. The ASM module also proved essential for reducing semantic confusion.
Qualitative Analysis: Visual comparisons showed that while baseline models often misclassify fine-grained actions (e.g., predicting "holding" instead of "eating"), MoSA correctly identified the specific motion-guided semantics.

5. Significance

The MoSA framework represents a significant step forward in video understanding by bridging the gap between low-level motion dynamics and high-level semantic reasoning.

Fine-Grained Understanding: It moves beyond static object detection to understand how objects interact, which is crucial for applications like autonomous driving, video question answering, and robotic manipulation.
Robustness to Imbalance: By effectively handling long-tail distributions, MoSA makes DSGG more practical for real-world scenarios where rare but critical events must be detected.
Multimodal Integration: The successful integration of visual motion cues with textual semantic priors demonstrates a powerful paradigm for resolving ambiguity in complex dynamic scenes.