💻 computer science

AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generation

AlignVid is a training-free method that enhances semantic fidelity in text-guided image-to-video generation by employing attention scaling modulation and guidance scheduling to counteract visual dominance, accompanied by the introduction of the OmitI2V benchmark for rigorous evaluation.

Original authors: Yexin Liu, Wen-Jie Shu, Zile Huang, Haoze Zheng, Yueze Wang, Jingjin Zhu, Manyuan Zhang, Ser-Nam Lim, Harry Yang

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yexin Liu, Wen-Jie Shu, Zile Huang, Haoze Zheng, Yueze Wang, Jingjin Zhu, Manyuan Zhang, Ser-Nam Lim, Harry Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Visual Anchor" Effect

Imagine you are a director trying to film a scene based on a script. You have a reference photo on the set (the "Image") and a script telling you what to do (the "Text Prompt").

In current AI video generators, the reference photo is like a heavy anchor. It's so visually strong and detailed that the AI gets "stuck" on it. Even if your script says, "Add a dragon to the sky" or "Make the person disappear," the AI ignores the script. It just keeps replaying the original photo because the visual details of the photo are shouting so loudly that the text instructions get drowned out.

The authors call this "Visual Dominance." The AI is so focused on what it sees that it forgets what it's supposed to do.

The Discovery: Blurring the Photo Helps (But Looks Bad)

The researchers ran a small experiment. They took a sharp, clear photo and gave it a Gaussian blur (making it fuzzy) before feeding it to the AI.

Surprisingly, this worked! When the photo was fuzzy, the AI actually listened to the text instructions. It added the dragon or removed the person.

Why? The blur removed the "noise" and distracting details of the photo. This forced the AI to stop staring at the pixels and start listening to the script.
The Catch: Blurring the input image ruins the final video quality. The video ends up looking fuzzy and low-quality. The researchers asked: Can we get the benefit of the blur (listening to the text) without actually blurring the image?

The Solution: AlignVid (The "Volume Knob" for Attention)

The answer is AlignVid. Instead of blurring the image before it enters the AI, AlignVid tweaks the AI's brain while it is thinking.

Think of the AI's attention mechanism like a sound mixing board with different volume knobs for different inputs:

The Image Channel: Very loud right now.
The Text Channel: Very quiet.
The Video Channel: Just right.

AlignVid acts as a smart volume knob. It doesn't change the image file itself. Instead, it turns down the volume on the image's details and turns up the volume on the text instructions inside the AI's processing.

How it works (The Two-Step Dance):

Attention Scaling Modulation (ASM): This is the main volume knob. It mathematically "sharpens" the AI's focus. Imagine the AI is looking at a crowd of people (tokens). Before, it was looking at everyone equally. ASM makes the AI focus intensely on the specific people mentioned in the script (the text) and ignore the background noise (the image details). This is described as reducing "entropy" (confusion) and making the AI's attention more decisive.
Guidance Scheduling (GS): This is the timing switch. If you turn up the text volume too much, too early, or for too long, the video might look weird or glitchy. GS is like a director saying, "Okay, turn up the text volume only during the first part of the scene where we decide what happens, then turn it back down so the final picture looks beautiful." It applies the fix only when and where it is needed.

The Result: A New Benchmark (OmitI2V)

To prove this works, the team didn't just guess; they built a test called OmitI2V.

Imagine a test with 367 different scenarios.
Some ask the AI to Add something (e.g., "Put a cat on the table").
Some ask it to Delete something (e.g., "Make the car vanish").
Some ask it to Modify something (e.g., "Change the red car to blue").

They found that without AlignVid, top AI models often failed these tests, ignoring the instructions. With AlignVid, the models followed the instructions much better, successfully adding, removing, or changing objects, all without needing to retrain the AI or slowing it down significantly.

Summary

The Problem: AI video generators are too obsessed with the starting image and ignore instructions to change it.
The Insight: Blurring the image helps the AI listen, but ruins the picture.
The Fix (AlignVid): A "training-free" method that acts like an internal volume knob. It turns down the image's "shout" and turns up the text's "whisper" inside the AI's brain, but only at the right moments.
The Outcome: The AI now follows instructions to add, remove, or change objects in videos much better, while keeping the video looking sharp and high-quality.

Note: The paper explicitly states this is a method for Image-to-Video generation and Image Editing. It does not claim to be used for medical diagnosis, clinical applications, or other specific real-world deployments beyond creative generation.

Technical Summary: AlignVid

1. Problem Statement

Text-guided image-to-video (TI2V) generation has advanced significantly in motion extrapolation but struggles with semantic fidelity when prompts require substantial edits to a reference image (e.g., adding, removing, or modifying objects). The paper identifies this failure mode as semantic negligence, where the model preserves the reference image's semantics rather than executing the text-specified changes.

Through empirical analysis, the authors attribute this to visual dominance: the high-fidelity reference image induces severe attention dispersion (high entropy) in the model's attention mechanism. This dominance suppresses the text modality and inhibits the model's ability to incorporate new semantic information or generate necessary motion. A pilot study revealed that applying Gaussian blur to the input image alleviates this dominance, reducing attention entropy and improving semantic adherence, but at the cost of visual quality. The core research question is whether this entropy-reduction effect can be achieved by regulating the model's internal attention distribution without degrading the input image.

2. Methodology: AlignVid

The authors propose AlignVid, a training-free intervention that re-calibrates the model's internal attention distribution based on an energy-based perspective of attention. The method consists of two primary components:

A. Attention Scaling Modulation (ASM)

Drawing on the theoretical link between attention scaling and temperature control in softmax functions, ASM rescales query ( $Q$ ) or key ( $K$ ) embeddings to sharpen the attention landscape.

Mechanism: Scaling $Q$ or $K$ by a factor $\gamma > 1$ is mathematically equivalent to increasing the inverse temperature ( $\alpha$ ) of the softmax.
Effect: This reduces the entropy of the attention distribution within the conditioning tokens (text and image), forcing the model to concentrate attention on a smaller set of high-logit, semantically relevant tokens.
Implementation:
- In MMDiT architectures (where conditioning is injected via queries), ASM modulates $Q$ .
- In DiT architectures (standard cross-attention), ASM modulates $K$ .
- Two variants are proposed: Scalar Scaling (fixed $\gamma$ ) and Energy-based Scaling (adaptive $\gamma$ based on logit sharpness).

B. Guidance Scheduling (GS)

To prevent indiscriminate application of ASM from degrading visual quality, the authors introduce a scheduling mechanism that gates the modulation at two levels:

Block-Level Guidance Scheduling (BGS): Modulation is applied only to "foreground-sensitive" transformer blocks. These blocks are identified via a lightweight calibration process using PCA and a grounding model to determine which layers focus heavily on foreground tokens.
Step-Level Guidance Scheduling (SGS): Modulation is activated only during specific intervals of the denoising trajectory. The authors find that applying guidance during early denoising steps yields the best balance, as these steps determine global semantic alignment, whereas later steps refine visual details.

3. Key Contributions

Problem Analysis & Pilot Study: The paper formalizes the semantic fidelity challenge in TI2V, empirically linking it to visual dominance and high attention entropy. It demonstrates that attention concentration is essential for liberating textual signals from visual redundancy.
AlignVid Framework: A training-free method that modulates attention via ASM and GS. It improves semantic alignment with negligible computational overhead and minimal aesthetic impact by re-weighting the modal hierarchy within the attention mechanism.
OmitI2V Benchmark: The authors introduce a comprehensive benchmark containing 367 human-annotated samples specifically designed to evaluate prompt adherence in three scenarios: modification, addition, and deletion. Unlike existing benchmarks that focus on general video quality, OmitI2V targets fine-grained semantic compliance using a Visual Question Answering (VQA) protocol.

4. Experimental Results

Extensive experiments on the OmitI2V benchmark and other standard benchmarks (VBench, GenEval, ImgEdit) demonstrate the efficacy of AlignVid:

Semantic Fidelity: AlignVid consistently improves semantic alignment scores across all edit types (modification, addition, deletion) when applied to baseline models like FramePack, FramePack-F1, and Wan2.1. For instance, on FramePack, it improved addition accuracy by +4.58% and modification accuracy by +3.23%.
Visual Quality: While there is a slight trade-off, the decrease in aesthetic quality is minimal (e.g., -0.37 to -1.49 points on aesthetic scores) compared to the substantial gains in semantic fidelity and dynamic degree.
Generalization: The method shows positive transferability to Text-to-Image (T2I) generation (OmniGen2) and Text-to-Video (T2V) tasks, improving performance on GenEval and VBench respectively. It also enhances image editing tasks on the ImgEdit benchmark.
Efficiency: The method introduces negligible inference overhead (less than 0.1% increase in runtime) as it only modifies attention weights without requiring additional training or complex modules.
Comparison with CFG: AlignVid complements Classifier-Free Guidance (CFG); applying AlignVid on top of CFG yields further improvements, offering a practical advantage by requiring no additional training.

5. Significance and Claims

The paper claims that AlignVid provides a tunable semantic–quality trade-off by directly regulating the model's internal attention distribution, mimicking the beneficial effects of input-level perturbations (like blurring) without corrupting the visual input.

The authors position their work as a diagnostic and corrective tool for the prevalent issue of semantic negligence in current TI2V models. By establishing a theoretical link between attention entropy and semantic selection, they offer a principled, lightweight approach to enhancing controllability in generative video models. The introduction of the OmitI2V benchmark is highlighted as a critical contribution for rigorously evaluating fine-grained edit compliance, a gap the authors argue exists in current evaluation suites.

The paper maintains a modest tone regarding its impact, noting that while improved controllability benefits creative applications, it necessitates responsible deployment practices to mitigate risks associated with generating persuasive synthetic content.