💻 computer science

SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models

SARA (Semantically Adaptive Relational Alignment) enhances video diffusion models by introducing a text-conditioned saliency mechanism that dynamically routes token-relation distillation supervision toward prompt-relevant subject interactions, thereby significantly improving text alignment and motion quality compared to existing methods.

Original authors: Jiesong Lian, Zixiang Zhou, Ruizhe Zhong, Yuan Zhou, Qinglin Lu, Rui Wang, Long Hu, Yixue Hao, Baoru Huang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiesong Lian, Zixiang Zhou, Ruizhe Zhong, Yuan Zhou, Qinglin Lu, Rui Wang, Long Hu, Yixue Hao, Baoru Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Distracted Artist"

Imagine you hire a talented artist to paint a scene based on your description: "A red dog chasing a blue cat through a park."

The artist is great at painting realistic fur, smooth grass, and moving water. However, they have a bad habit of getting distracted. Sometimes they paint a green dog, sometimes they forget the cat entirely, and sometimes they paint the dog and cat standing still instead of chasing each other. They are following the general vibe of your request but missing the specific details.

In the world of AI video generation (Video Diffusion Models), this is exactly what happens. The AI can make beautiful, smooth videos, but it often fails to follow the specific instructions in the prompt. It might drop an object, mix up colors, or ignore how two things interact.

The Old Solution: The "Blind Spot"

Researchers tried to fix this by teaching the AI to look at a "master reference" (a frozen visual model) while it paints. They told the AI: "Match the relationship between every single pixel in your video to the relationships in the master reference."

The Flaw: This is like telling the artist to pay equal attention to the dog, the cat, the sky, the grass, and the dirt on the ground.

The prompt only cares about the dog and the cat.
The dirt and the sky (the "background") are just filler.
By forcing the AI to study the dirt and sky just as hard as the dog and cat, the AI wastes its brainpower on things that don't matter. It's like studying a textbook where 70% of the pages are about the color of the ink, and only 30% are about the actual lesson.

The New Solution: SARA (The "Smart Spotlight")

The authors propose SARA (Semantically Adaptive Relational Alignment). Think of SARA as a smart spotlight that tells the AI exactly where to focus its attention.

Instead of treating every part of the video equally, SARA asks: "What did the user actually ask for?"

The "Spotlight" (Text-Conditioned Saliency):
Before the AI starts painting the video, SARA reads the prompt and uses a "spotlight" to highlight the important parts. If you say "red dog," the spotlight shines bright on the dog and the space around it. If you mention a "blue cat," the spotlight moves there. It ignores the empty sky or the generic grass unless the prompt specifically mentions them.
The "Traffic Controller" (Pair-Routing):
In the old method, the AI tried to learn how the dog relates to the cat, the cat relates to the grass, and the grass relates to the sky.
SARA acts like a traffic controller. It says:
- Yes: Learn how the Dog relates to the Cat (Subject-to-Subject).
- Yes: Learn how the Dog relates to the Grass (Subject-to-Background, because the dog is on the grass).
- No: Stop worrying about how the Grass relates to the Sky (Background-to-Background). That's just noise.
SARA uses a simple logic rule (an "OR" gate): If either the dog or the cat is in the spotlight, the AI pays attention to their relationship. This ensures the AI focuses its energy on the things you actually care about.

How It Works (The Two-Step Process)

Step 1: Training the Spotter (The "Lighting Crew")
First, the researchers train a small, lightweight assistant. This assistant learns to look at a video and a text description and figure out exactly which parts of the video match the words.

It uses a tool called SAM 3.1 (a super-accurate object detector) to draw masks around objects.
It learns to say, "When the text says 'red dog,' this specific patch of pixels is the dog."
Crucially, it learns to handle multiple objects at once without getting confused.

Step 2: The Main Show (The "Director")
Once the "Spotter" is trained, it is frozen (it doesn't change anymore). Now, the main video AI starts its training.

As the main AI tries to generate the video, the "Spotter" shines its light on the important parts.
The main AI is only forced to learn the relationships between the highlighted parts.
This makes the learning process much more efficient. The AI stops wasting time on the background and starts mastering the specific interactions you asked for.

The Results: Why It Matters

The paper tested this on a popular open-source video model called Wan2.2. They compared SARA against:

Standard Training: Just letting the AI learn normally.
Old Methods: The "Blind Spot" approach that treats everything equally.

The Outcome:
SARA produced videos that were much better at following instructions.

Better Accuracy: If you asked for a "red dog," SARA actually made a red dog. The others often made a brown one or forgot the dog.
Better Motion: The interactions (like the dog chasing the cat) were smoother and more logical.
User Preference: In blind tests where humans couldn't see which AI made the video, people consistently preferred the SARA videos over the others.

Summary

Think of SARA as upgrading the AI from a student who highlights the whole page (wasting time on irrelevant details) to a student who uses a highlighter to mark only the key terms in the textbook. By focusing its "study time" (computational power) only on the relationships that matter to the prompt, SARA creates videos that are not just pretty, but actually do what you asked them to do.

Technical Summary: SARA (Semantically Adaptive Relational Alignment)

Problem Statement

Recent Video Diffusion Models (VDMs) have achieved high visual fidelity and temporal coherence but struggle with fine-grained semantic controllability. They frequently drop entities, mis-bind attributes to incorrect subjects, or weaken interactions specified in the prompt. While representation-alignment objectives like VideoREPA and MoAlign improve text following by distilling spatio-temporal token relations from a frozen Visual Foundation Model (VFM), they suffer from a supervision allocation problem.

VideoREPA weights all token pairs equally ( $O(N^2)$ budget), regardless of the prompt. In multi-entity scenes, this results in roughly 27% of the supervision budget being wasted on background-background pairs (which the prompt never references), while subject-subject pairs (critical for inter-entity relations) receive only ~23%.
MoAlign attempts to fix this by compressing the VFM target into a motion-supervised subspace, effectively acting as an AND router that only supervises moving subject-subject pairs. However, optical flow is a noisy proxy for saliency (failing under occlusion, fast motion, or low texture) and ignores static subjects and subject-background pairs, which are essential for grounding entities in their scenes.

The core challenge is how to allocate a fixed $O(N^2)$ pair-budget to concentrate supervision on prompt-relevant relations (both subject-subject and subject-background) rather than background filler.

Methodology: SARA

SARA proposes a two-stage framework that retains the Token-Relation Distillation (TRD) form of VideoREPA but introduces a text-conditioned saliency to dynamically route supervision.

Stage 1: Training a Text-Conditioned Saliency Aligner

A lightweight aligner is trained offline to predict a continuous saliency map $M_p$ conditioned on the text prompt.

Backbones: Uses frozen V-JEPA (visual), SAM 3.1 (segmentation), and Qwen3-VL-Embedding (text).
Architecture: A cross-attention module fuses visual tokens with text embeddings, followed by a saliency head (MLP + sigmoid) and a visual projector for contrastive learning.
Supervision Strategy:
1. Entity-Separated Supervision: Instead of a single global mask, the aligner is trained on $K+2$ forwards per video: $K$ per-entity pairs (caption + binary mask), one combined-entity pair, and one background pair. This prevents the model from collapsing into a generic foreground prior.
2. InfoNCE Regularizer: Aligns the fused visual features with the text hidden states in embedding space to prevent the saliency from collapsing onto the most dominant subject, preserving cross-entity contrast.
Output: A continuous saliency map $M_p \in [0, 1]$ indicating the relevance of each token to the prompt.

Stage 2: Saliency-Routed TRD

During the continual training of the VDM (specifically Wan2.2), the frozen Stage 1 aligner generates saliency maps for the full video caption. These maps are fused into the TRD loss via a pair-routing operator.

Pair-Routing Operators: SARA defines fuzzy-logic relaxations of Boolean operations to map token saliencies ( $w_i, w_j$ $w_{i}, w_{j}$ ) to pair weights ( $W_{ij}$ $W_{ij}$ ):
- AND: $w_i w_j$ (Retains only subject-subject pairs; similar to MoAlign's effect).
- XOR: $|w_i - w_j|$ (Retains only boundary pairs).
- OR (Default): $w_i + w_j - w_i w_j$ . This is the core innovation. It retains subject-subject pairs and subject-background pairs, routing supervision away from background-background pairs.
Loss Function: The standard TRD loss is modified to be a weighted average where the weights are determined by the OR operator. This reallocates the supervision budget from background noise to semantically relevant relations without changing the VFM target or the number of trainable parameters in the diffusion path.

Key Contributions

Conceptual: Reframes semantic adaptation for VDMs as a pair-routing problem on top of TRD. It formalizes MoAlign as an implicit AND router and proposes a text-supervised OR router that explicitly includes subject-background grounding.
Methodological: Introduces a text-conditioned saliency aligner trained with per-entity SAM masks and an InfoNCE regularizer. This allows the system to dynamically decide which token pairs carry supervision based on the prompt, rather than static motion cues.
Empirical: Demonstrates that SARA consistently outperforms SFT, VideoREPA, and MoAlign on the Wan2.2 backbone across multiple evaluation protocols.

Results

Experiments were conducted on a 500K clip multi-entity corpus using the Wan2.2 high-noise transformer. Evaluation included a 13-dimension VLM rubric, public VBench-1.0/2.0 suites, and a blind user study.

VLM Rubric: SARA is the only method to improve both Text Alignment (TA) and Motion Quality (MQ) simultaneously compared to the pretrained baseline. It outperforms the strongest baseline (MoAlign) on all dimensions.
VBench: SARA achieves the highest aggregate scores on both VBench-1.0 (Semantic: +0.90 over VideoREPA) and VBench-2.0 (Final: +0.38 over MoAlign).
User Study: In a blind pairwise study, annotators preferred SARA over all baselines, with the largest margin against the pretrained model.
Ablations: Removing the saliency head or the InfoNCE regularizer causes significant performance drops. Replacing the OR router with AND or XOR also degrades performance, confirming that the inclusion of subject-background pairs is critical.

Significance and Claims

The paper claims that SARA addresses the "dilution" of supervision signals in current representation-alignment methods. By shifting the allocation of the $O(N^2)$ budget from a geometry- or motion-driven heuristic to a prompt-driven heuristic, SARA ensures that the model learns to bind attributes and interactions specifically where the text indicates.

The authors note that while SARA improves prompt following, it inherits the limitations of the base model's training protocol. Specifically, because only the high-noise expert of the Wan2.2 MoE is updated, there is a slight, consistent drop in "low-noise" rendered qualities (e.g., human fidelity, temporal consistency) compared to the frozen pretrained model. However, SARA exhibits the smallest drop among all continual-training methods, suggesting its targeted updates are less disruptive to the frozen low-noise expert than uniform updates.

The paper concludes that SARA provides a more efficient and semantically aware mechanism for aligning video diffusion models, achieving state-of-the-art results in fine-grained text following without requiring additional inference-time conditions or architectural changes to the diffusion backbone.