💻 computer science

DynamicRad: Content-Adaptive Sparse Attention for Long Video Diffusion

DynamicRad is a content-adaptive sparse attention framework that leverages a radial locality prior and an offline Bayesian optimization-driven semantic motion router to achieve significant inference speedups (1.7×–2.5×) and high sparsity (>80%) in long video diffusion models while preserving or enhancing generation quality.

Original authors: Yongji Long, Shijun Liang, Jintao Li, Yun Li

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yongji Long, Shijun Liang, Jintao Li, Yun Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to watch a 10-hour movie, but your brain (the computer) can only hold a few seconds of it in its "active memory" at a time. To understand the whole story, your brain has to constantly look back and forth between the beginning, middle, and end.

In the world of AI video generation, this "looking back and forth" is called Attention. The problem is that for long videos, the AI tries to look at every single frame against every other frame. It's like trying to read every page of a 1,000-page book to understand just one sentence. This takes forever and uses up all the computer's energy.

DynamicRad is a new "smart reading strategy" that solves this. Here is how it works, broken down into simple concepts:

1. The Problem: The "Brute Force" Approach

Current AI video makers are like a paranoid librarian. To find a specific fact, they check every single book on every single shelf, even if the book is clearly about a different topic.

The Result: It's accurate, but incredibly slow. Making a long video can take hours.

2. The Old Solution: The "Rigid Fence"

Some previous methods tried to speed things up by building a rigid fence. They said, "We will only look at the books within 5 feet of the current one."

The Flaw: This works great for a quiet library (a video of a sleeping cat). But if the scene changes to a chaotic car chase (fast motion), the "fence" is too small. The AI misses the car that jumped in from 20 feet away, and the video looks glitchy or broken.

3. The New Solution: DynamicRad (The "Smart Scout")

DynamicRad is like hiring a Smart Scout who knows exactly when to be strict and when to be flexible. It uses two main tricks:

A. The "Motion Sensor" (Semantic Motion Router)

Before the AI starts generating the video, it reads the text prompt (e.g., "A slow sunset" vs. "A fast drone race").

Low Motion (The Sunset): The Scout says, "Okay, nothing is moving much. We can be lazy! Let's only look at the immediate neighbors." This saves massive amounts of time.
High Motion (The Drone Race): The Scout says, "Whoa, things are moving fast! We need to keep our eyes on the horizon. Let's look further back and forward to make sure we don't miss the action."

B. The "Two Modes" (Static vs. Dynamic)

The system has two gears it can switch between instantly:

Speed Gear (Static-Ratio): Used when you just want the video done fast. It cuts out 80% of the "looking" work by following a strict rule. It's like skimming a book, reading only the first and last sentence of every paragraph.
Quality Gear (Dynamic-Threshold): Used when you want it to look perfect. It looks at the "importance" of every frame. If a frame is boring, it ignores it. If a frame is exciting (like an explosion), it keeps it, even if it's far away in the timeline.

4. The "Offline Training" (The Cheat Sheet)

Usually, figuring out the best way to watch a video takes time while the video is being made. DynamicRad does the homework beforehand.

Imagine a chef who tastes a thousand different spice blends before the customer orders, so when the order comes in, they just grab the perfect jar instantly.
DynamicRad uses a "Cheat Sheet" (created by a mathematical optimizer called Bayesian Optimization) to know exactly how much to "skip" based on the type of video. This means there is zero delay when the video starts generating.

5. The "Safety Net" (Mask-Aware LoRA)

Sometimes, skipping too much information makes the video look weird (like a character's face flickering).

DynamicRad adds a tiny "patch" (called LoRA) that acts like a safety net. It gently fixes the small errors caused by skipping, ensuring the video stays smooth and coherent, even when the AI is being very aggressive about skipping frames.

The Bottom Line

DynamicRad is like upgrading from a car that drives at a constant, slow speed to a smart sports car.

On a straight, empty road (slow video), it cruises at 200 mph, ignoring everything unnecessary.
On a winding, dangerous track (fast video), it slows down just enough to take the corners perfectly, ensuring you don't crash.

The Result: The AI can now generate long, high-quality videos 2 to 2.5 times faster without losing the quality, making it possible to create movies that used to take days, now done in hours.

1. Problem Statement

Video diffusion models (e.g., HunyuanVideo, Wan2.1) face a critical scalability bottleneck: self-attention complexity grows quadratically ( $O(N^2)$ ) with the number of spatiotemporal tokens. This makes generating long, high-resolution videos computationally prohibitive due to memory bandwidth and compute constraints.

Existing solutions fall into two categories with distinct limitations:

Static Sparse Patterns: (e.g., Radial Attention) are hardware-friendly but rigid. They fail to adapt to complex motion dynamics, often losing critical long-range dependencies in fast-moving scenes.
Dynamic Sparsification: (e.g., SVG, STA) adapts to content but introduces significant online overhead (searching for optimal tokens per step) or requires expensive retraining, making them unstable or slow at scale.

The core challenge is achieving high sparsity (>80%) for efficiency without sacrificing the temporal coherence and visual fidelity required for long-horizon video generation.

2. Methodology: DynamicRad

The authors propose DynamicRad, a unified sparse-attention paradigm that combines structured sparsity with content adaptivity. The framework operates on three main pillars:

A. Unified Candidate Set & Dual-Mode Selection

DynamicRad utilizes a shared, hardware-friendly block-sparse candidate set derived from a radial locality prior (attention mass decays with spatiotemporal distance). It offers two mutually exclusive selection modes:

Static-Ratio Mode (Throughput-First):
- Designed for speed.
- Uses a deterministic retention ratio based on temporal distance ( $t$ ).
- No content-dependent scoring overhead; simply samples a fixed percentage of tokens from the candidate set.
Dynamic-Threshold Mode (Quality-First):
- Designed for fidelity.
- Uses pre-softmax attention logits as lightweight importance proxies.
- Applies a global threshold to filter tokens, retaining only high-scoring pairs.
- Includes safeguards: if a block is empty after thresholding, the top-scoring pair is retained to prevent continuity breaks.

B. Offline Bayesian Optimization (BO) & Semantic Motion Router

To avoid the overhead of online search, DynamicRad decouples configuration from inference:

Offline BO Pipeline: A Bayesian Optimizer (using TPE) searches for optimal hyperparameters (decay factor $\gamma$ , block size $B_s$ , thresholds $\theta$ , long-range factor $\lambda$ ) on a physics-based proxy task. The objective is to minimize the Mean Squared Error (MSE) between the sparse approximation and dense attention on a simulated feature drift process. This search completes in <15 minutes.
Semantic Motion Router: A lightweight 2-layer MLP maps text prompt embeddings (pre-computed) to a motion regime (Low, Mid, High).
- Low Motion (e.g., "reading"): Triggers aggressive static masking.
- High Motion (e.g., "FPV drone"): Triggers dynamic thresholding with denser long-range connections.
- Overhead: <2ms per video, amortized to ~0ms per layer.

C. Mask-Aware LoRA

To compensate for information loss caused by aggressive sparsity, the authors introduce a Mask-Aware LoRA module.

It fine-tunes only the projection layers (Q/K) sensitive to the sparsity pattern while freezing the backbone.
This acts as a "denoising" mechanism, restoring temporal consistency and fine-grained details without retraining the entire model.

3. Key Contributions

Unified Framework: Bridges the gap between rigid static priors (Radial Attention) and high-overhead dynamic methods (SVG) by introducing a shared candidate set with dual selection modes.
Training-Free Adaptivity: Achieves content-aware sparsity without auxiliary networks or per-step routing inference, relying instead on an offline BO pipeline and a lightweight semantic router.
Robust Deployment: Introduces a Mask-Aware LoRA strategy that significantly improves temporal coherence under aggressive sparsity, a common failure point in other sparse methods.
Efficiency-Optimized Proxy: Demonstrates that optimizing a lightweight physics-based proxy (feature drift) effectively translates to improved perceptual quality in final video generation.

4. Experimental Results

The method was evaluated on HunyuanVideo-241f and Wan2.1-14B across various resolutions and sequence lengths.

Speedup: Achieved 1.7× to 2.54× inference speedups compared to dense baselines.
- Example: On HunyuanVideo-241f, DynamicRad reduced latency from 824s (Dense) to 324s (Static Mode), a 2.54× improvement.
Sparsity: Maintained effective sparsity rates of >80% (up to 86.8% in static mode).
Quality:
- Dynamic Mode: Surprisingly, in long-sequence settings, the dynamic mode outperformed the dense baseline in quality metrics (e.g., VisionReward: 0.131 vs. 0.116 for HunyuanVideo-241f). The authors attribute this to the "denoising" effect of pruning irrelevant OOD tokens.
- Temporal Coherence: With Mask-Aware LoRA, VBench Structural Consistency scores reached 0.977, matching or exceeding dense baselines even under high sparsity.
Overhead: The routing and scoring mechanisms added negligible latency (<0.5ms per layer), confirming the efficiency of the design.

5. Significance

DynamicRad represents a significant step forward in making long-video generation practical for real-world deployment.

Paradigm Shift: It moves away from the trade-off between "fast but rigid" and "adaptive but slow" by proving that offline optimization combined with semantic routing can achieve both.
Scalability: By reducing the quadratic attention cost to near-linear complexity (via logarithmic window decay and sparsity), it enables the generation of hundreds of frames on standard hardware clusters.
Generalizability: The method is model-agnostic (works on DiT and VAE backbones) and does not require retraining the massive diffusion backbone, making it highly applicable to existing video generation ecosystems.

In conclusion, DynamicRad successfully pushes the efficiency–quality Pareto frontier, demonstrating that high-speed, high-fidelity long-video generation is achievable through content-adaptive sparse attention.