🤖 machine learning

LVSA: Training-Free Sparse Attention for Long Video Diffusion

LVSA is a training-free, model-agnostic block-sparse attention mechanism that combines structured windows with rotating global anchors to significantly accelerate long-video diffusion inference and extend generation horizons while maintaining or improving video quality, alongside the introduction of VQeval for more accurate quality assessment.

Original authors: Gael Glorian, Ioannis Lamprou, Zhen Zhang, Yujie Yuan, Hongsheng Liu

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gael Glorian, Ioannis Lamprou, Zhen Zhang, Yujie Yuan, Hongsheng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to direct a movie using a very smart, but slightly overwhelmed, AI assistant. This assistant is great at making short clips, but when you ask it to make a long movie (hundreds of frames long), it starts to panic.

Here is the problem the paper solves, explained through a few simple stories:

1. The Problem: The "Overwhelmed Librarian"

Current video AI models work like a librarian who has to check every single book on every shelf to answer a single question.

The Cost: If you have 100 frames of video, the librarian has to read 100 books. If you have 1,000 frames, they have to read 1,000 books. But here's the catch: to make a good connection, they have to compare every book to every other book. The work doesn't just double; it explodes (quadratically). This makes long videos incredibly slow and expensive to generate.
The "Frozen" Glitch: When the librarian gets too tired (because the video is too long), they stop thinking creatively. They just start repeating the same page over and over. In video terms, the characters stop moving, the background stops changing, and the video becomes a "frozen" or "looping" mess.

2. The Solution: LVSA (The "Smart Tour Guide")

The authors introduce LVSA (Long Video Sparse Attention). Think of this as replacing the overwhelmed librarian with a Smart Tour Guide.

Instead of reading every single book in the library, the guide uses a clever strategy:

The Local Window: For the current scene, the guide only looks at the immediate surroundings (the "local window"). This keeps the details sharp and the action smooth.
The Global Anchors: To remember the big picture, the guide picks a few "landmarks" (global anchors) scattered throughout the movie. They check these landmarks periodically to make sure the story hasn't drifted off the rails.
The Rotating Shift: Here is the magic trick. In the old method, the guide always checked the same landmarks. This caused the "frozen" glitch because the guide got bored of those specific spots. LVSA rotates the landmarks. At one moment, it checks the start of the movie; at the next, it checks a spot slightly later. This keeps the guide fresh and ensures the whole movie feels alive, not just a static loop.

Best of all: The guide doesn't need to be retrained. You can just hand this new strategy to the existing AI, and it works immediately.

3. The Results: Faster, Longer, and Better

The paper tested this "Smart Tour Guide" on three different powerful AI models (Wan and HunyuanVideo) and found:

Speed: It made generating long videos 3 times faster (sometimes even more) than the old method.
- Analogy: If the old way took 50 minutes to make a long clip, the new way does it in about 16 minutes.
Feasibility: Some videos were simply impossible to make before because they required too much computer memory (the "librarian" ran out of desk space). LVSA shrinks the memory needed so much that these long videos can now be made on a single standard computer chip.
Quality: The videos are much more dynamic. The characters move naturally instead of freezing.
- The "Freezing" Test: The authors created a new scoring tool called VQeval. Old scoring tools were like a teacher who gave a "A+" to a student who just stared at the wall because they were "consistent." VQeval is a stricter teacher that gives a "F" to the frozen video and a "A" to the one with real movement. LVSA gets the "A."

4. Real-World Testing

The team didn't just test this on one type of computer. They showed it works on:

GPUs: The standard powerful chips used for AI.
NPUs: Specialized chips found in some newer devices, proving this method is flexible enough to run on different hardware.

Summary

LVSA is a free, plug-and-play upgrade for video AI. It stops the AI from getting "tired" and freezing up during long videos. It does this by having the AI focus on the immediate action while occasionally checking rotating "landmarks" to keep the story moving. The result is videos that are faster to create, fit on smaller computers, and actually look like moving pictures instead of static slides.

Technical Summary: LVSA: Training-Free Sparse Attention for Long Video Diffusion

1. Problem Statement

Video Diffusion Transformers (DiTs), such as Wan and HunyuanVideo, face two critical bottlenecks when generating long videos:

Computational Cost: Standard dense self-attention scales quadratically ( $O(N^2)$ ) with the sequence length (number of frames). This makes inference for long videos prohibitively expensive.
Quality Degradation: Beyond the training horizon (e.g., 81 frames for Wan, 129 for HunyuanVideo), dense attention causes the model to converge to near-static, "frozen," or looping outputs. This results in a loss of motion and temporal variation, rendering the video low quality.

Existing solutions are insufficient: some require costly retraining, while others (like sliding window or radial attention) fail to eliminate long-range temporal repetition artifacts. Conversely, extrapolation methods that preserve quality (e.g., RIFLEx, UltraViCo) often fail to reduce computational costs, as they still rely on dense attention matrices.

2. Methodology: Long Video Sparse Attention (LVSA)

The authors propose LVSA, a training-free, model-agnostic block-sparse attention mechanism designed to balance compute efficiency with temporal coherence.

Core Mechanism

LVSA restricts the set of frames $A(t)$ that a query frame $t$ attends to, replacing the full set $\{0, \dots, T-1\}$ with a union of two components:
$A(t) = G \cup W(t)$

Rotating Global Anchors ( $G$ ):
- Instead of a fixed grid of global frames (which causes bias where intermediate frames are only seen locally), LVSA employs rotating periodic global frames.
- At each denoising step $s$ , the set of global frames $G_s$ shifts by $s \pmod{T_{per}}$ .
- Over $T_{per}$ consecutive steps, every frame serves as a global anchor exactly once. This eliminates fixed-grid bias and prevents the systematic impoverishment of intermediate frame representations, thereby reducing long-range temporal artifacts like repetition and identity drift.
Expanded Adaptive Windows ( $W(t)$ ):
- Each frame attends to a local temporal window of size $W$ .
- To ensure a constant attention budget per query frame, the authors introduce expanded window bounds. If the local window overlaps with global frames, the window is dynamically extended to compensate for the overlap, ensuring $|A(t)| \approx C$ (a target budget) for all frames.
- The target budget $C$ is typically set to the model's training horizon (e.g., 21 frames for Wan 2.1 1.3B), ensuring the inference cost matches the training cost.

Complexity

By fixing the number of attended frames to $C$ , the complexity of attention per query becomes $O(C \cdot P \cdot d)$ , leading to a total complexity of $O(T \cdot C \cdot P^2 \cdot d)$ . This is asymptotically linear in the number of frames $T$ , enabling scalable long-video generation.

Evaluation Benchmark: VQeval

The authors introduce VQeval, a custom evaluation tool designed to properly score "loopy" or static video failures. Unlike state-of-the-art evaluators like VBench-Long, which reward inter-frame similarity (and thus inadvertently score frozen videos highly), VQeval explicitly penalizes lack of motion and looping artifacts through its "dynamic quality" and "loop quality" dimensions.

3. Key Contributions

LVSA Algorithm: A novel, training-free sparse attention pattern combining rotating global anchors and expanded adaptive windows to remove fixed-grid bias and reduce compute.
VQeval Benchmark: A new metric specifically calibrated to detect and penalize the static/looping failure modes common in long-video extrapolation, addressing the shortcomings of existing metrics.
Cross-Architecture Validation: Experimental proof of efficacy across three distinct DiT architectures (Wan 2.1 1.3B, Wan 2.1 14B, and HunyuanVideo 1.5) and hardware platforms (GPUs and NPUs).
Open-Source Implementation: The method is released as a plugin for popular open-source platforms.

4. Experimental Results

Computational Efficiency

Speedups: On an 80GB GPU, LVSA combined with a FlashInfer kernel achieves significant speedups compared to dense attention:
- 3.17× on Wan 2.1 1.3B (6× horizon).
- 2.98× on Wan 2.1 14B (6× horizon).
- 3.33× on HunyuanVideo 1.5 (1.5× horizon).
Memory Feasibility: LVSA enables generation at horizons where dense attention fails due to Out-Of-Memory (OOM) errors. For example, HunyuanVideo 1.5 generation at a 2× horizon (257 frames) is infeasible with dense attention on a single 80GB GPU but runs successfully with LVSA (peak memory ~60 GB).
NPU Performance: On NPUs, LVSA achieves speedups of 2.71× (Wan 2.2 A14B) and 3.24× (Wan 2.1 1.3B) at a 6× horizon.

Video Quality

Training Horizon: At the native training horizon (1×), LVSA is quality-neutral compared to dense attention across all metrics.
Extended Horizons: Beyond the training horizon, LVSA significantly outperforms dense attention.
- On Wan 2.1 1.3B (6× horizon), LVSA-FI scores +12.1 on the VQeval composite compared to dense attention, which collapses to near-static output.
- VBench-Long vs. VQeval: While VBench-Long scores often rise for dense attention at extended horizons (due to its bias toward static consistency), VQeval correctly identifies this as a quality failure. LVSA maintains genuine motion, resulting in higher VQeval scores.
Comparison to SOTA: LVSA outperforms RIFLEx and UltraViCo in both compute and quality:
- Speed: LVSA is up to 3.27× faster than UltraViCo and 2.41× faster than RIFLEx on Wan 2.1 1.3B.
- Quality: LVSA achieves the highest VQeval composite scores at all tested horizons, whereas RIFLEx is statistically indistinguishable from dense attention in terms of quality collapse, and UltraViCo incurs high latency costs.

5. Significance and Claims

The paper claims that LVSA successfully harnesses the tradeoff between compute and quality for long-video diffusion. By removing the fixed-grid bias inherent in static sparse patterns and utilizing a rotating global anchor strategy, LVSA prevents the "frozen" video artifacts that plague long-horizon generation.

The authors emphasize that their approach is training-free and model-agnostic, making it immediately applicable to existing DiT architectures without retraining. Furthermore, the introduction of VQeval highlights a critical flaw in current evaluation standards, arguing that metrics rewarding static consistency are misleading for long-video generation. The work demonstrates that it is possible to generate high-quality, long-duration videos on single GPUs and NPUs with linear scaling, overcoming the quadratic memory and compute barriers of dense attention.