💻 computer science

Token Radius Attention for Efficient Video Generation

Token Radius Attention (TRA) is a training-free framework that efficiently accelerates Video Diffusion Transformers by dynamically mapping query entropy to token-dependent attention radii, achieving significant speedups with minimal quality loss while retaining only 9-19% of attention interactions.

Original authors: Jiayu Chen, Zhikun Jiang, Maoliang Li, Jiayi Luo, Jiawei Yang, Zihao Zheng, Hengyi Zhang, Guojie Luo, Xiang Chen

Published 2026-08-04

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiayu Chen, Zhikun Jiang, Maoliang Li, Jiayi Luo, Jiawei Yang, Zihao Zheng, Hengyi Zhang, Guojie Luo, Xiang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a massive, moving mural on a wall that stretches for miles. To make the picture look real, you need to look at every single brushstroke you've made so far and decide how it connects to the one you are painting right now. In the world of artificial intelligence, this is exactly what "Video Diffusion Transformers" do. They are the digital artists creating stunning videos from scratch, but to make a video look smooth and realistic, the computer has to check every single pixel against every other pixel in the entire video. This is like trying to shake hands with every person in a stadium while you are standing in one spot; it's incredibly thorough, but it takes a huge amount of time and energy. Scientists call this "dense attention," and it's the reason why making high-quality AI videos is currently so slow and expensive. The big question researchers are asking is: Can we make the AI smarter about who it shakes hands with, so it skips the boring parts and focuses only on the important connections, without ruining the picture?

This paper introduces a clever new strategy called Token Radius Attention (TRA) to solve that problem. Instead of treating every part of the video the same way, the authors discovered that different parts of the video actually need different amounts of attention. Think of it like a party: some guests are deep in a serious conversation and only need to hear the people right next to them (low attention), while others are dancing wildly and need to keep an eye on the whole room (high attention). The researchers found that they could predict exactly how much "attention" each part of the video needs just by measuring how "confused" or "spread out" its current focus is. They call this measurement "entropy."

Using this insight, the team built a system that acts like a smart radius guard. For every tiny piece of the video (called a "token"), the system calculates a custom-sized circle around it. If the piece is focused and calm, the circle is small, and the AI only looks at its immediate neighbors. If the piece is chaotic or complex, the circle grows larger to include more neighbors. This happens automatically without the AI having to stop and rank every single possibility, which saves a massive amount of time. The result is a video generator that keeps 9% to 19% of the original connections but runs 1.56 to 2.05 times faster, producing videos that look just as good as the slow, heavy versions. The authors tested this on several different video-making models and found that it works consistently well, proving that you don't need to check everything to get a great result—you just need to check the right things.

Technical Summary: Token Radius Attention for Efficient Video Generation

1. Problem Statement

Video Diffusion Transformers (VDiTs) have become the dominant architecture for high-fidelity video generation, relying on dense 3D self-attention to capture complex spatial and temporal dependencies. However, dense attention incurs a quadratic computational cost ( $O(N^2)$ ) relative to the number of video tokens $N$ , creating a significant inference bottleneck for scaling and deployment.

Existing approaches to mitigate this cost employ sparse attention mechanisms, categorized primarily into head-level and block-level methods.

Head-level methods assign structured spatial or temporal patterns to individual attention heads.
Block-level methods select important query–key block pairs.

The Core Limitation: Both paradigms typically share a single computation budget (e.g., a fixed retention density or radius) across all query tokens within a head or block. This "shared-allocation" assumption conflicts with the intrinsic nature of self-attention, where the Softmax normalization is performed independently for each query. Consequently, queries with concentrated attention distributions (low entropy) waste computation if forced to attend to many keys, while queries with diffuse distributions (high entropy) may lose critical interactions if the shared budget is too small. This mismatch degrades generation quality or fails to achieve optimal efficiency.

2. Methodology: Token Radius Attention (TRA)

The authors propose Token Radius Attention (TRA), a training-free framework that realizes token-specific structured sparsity without explicit per-query key ranking. TRA operates on two key empirical insights derived from analyzing VDiT attention patterns.

Insight I: Token-Specific Attention Sparsity

The authors observe that the "retained density" (the fraction of top-ranked keys required to preserve a target attention mass) varies by orders of magnitude across queries within the same layer and head. Crucially, they find a strong log-linear relationship between this retained density and attention entropy.

Mechanism: High entropy indicates a diffuse attention distribution requiring a larger token budget, while low entropy indicates a concentrated distribution requiring fewer tokens.
Implementation: TRA uses an analytic approximation to map query entropy directly to a token-specific budget ( $\hat{B} \approx \tau \exp(H_i) / N$ ), eliminating the need for expensive per-query sorting or ranking.

Insight II: Token Radius Attention Pattern

The authors observe that dominant interactions for a query form query-centered circular neighborhoods in the spatial domain. The normalized attention probability decays approximately exponentially with 2D spatial distance.

Mechanism: Instead of selecting arbitrary top- $k$ keys, TRA converts the scalar token budget into a spatial radius.
Temporal Decay: To handle video sequences, the spatial radius is modulated by a temporal distance decay rule ( $\phi(\delta) = \exp(-\gamma \delta)$ ), creating a regular spatiotemporal support structure.

The TRA Pipeline

Entropy-to-Budget: During an initial dense "warm-up" phase, TRA computes attention entropy for each query. This entropy is mapped to a specific token budget.
Budget-to-Radius: The budget is converted into a query-specific base radius. This radius is then adjusted for each frame based on temporal distance to form a 2D disk mask.
Efficient Execution:
- Tile-Major Layout: Video tokens are rearranged into a tile-major order to ensure that spatially neighboring tokens (which fall within the radius) are contiguous in the sequence.
- Fused CUDA Kernels: A custom kernel fuses distance computation, radius comparison, block pruning, and token voting to convert the irregular token-level radius masks into regular block-sparse masks compatible with FlashInfer.
- Cross-Step Reuse: Entropy is computed only during the warm-up phase and reused for subsequent sparse steps, leveraging the observed stability of entropy patterns across denoising steps.

3. Key Contributions

Discovery of Token-Specific Sparsity: The paper uncovers that retained density varies dramatically across queries but follows a predictable log-linear relationship with attention entropy, and that dominant interactions follow a query-centered radius pattern.
Token Radius Attention (TRA): A training-free framework that transforms query-specific computation demands into regular spatiotemporal supports via an entropy-to-budget-to-radius pipeline, avoiding explicit key ranking.
System Co-Design: The authors co-design a radius-mask kernel and a fused entropy kernel to minimize overhead, enabling efficient execution on existing block-sparse backends.
Comprehensive Validation: TRA is validated across seven configurations of Wan2.1, Wan2.2, and HunyuanVideo (spanning 1.3B to 14B parameters) for both Text-to-Video (T2V) and Image-to-Video (I2V) tasks.

4. Experimental Results

TRA was evaluated against dense attention and three training-free sparse baselines (SVG, SVG2, and Radial) on the VBench benchmark suite.

Efficiency: TRA retains only 9%–19% of attention interactions. It achieves a speedup of 1.56× to 2.05× over dense models across all tested configurations. For example, on HunyuanVideo-13B, it achieves a 2.05× speedup for T2V and 2.02× for I2V.
Quality: TRA maintains competitive generation quality, often achieving the best VBench Overall scores among sparse methods. On Wan2.1-14B T2V, it nearly matches the dense attention score while providing a 1.75× speedup.
Fidelity: Qualitative results show TRA better preserves subject identity, structural integrity, and temporal consistency compared to other sparse methods, with fewer artifacts like flickering or distortion.
Ablation Studies: Removing the entropy-guided budget mapping or the radius masking (replacing it with 1D distance) significantly degrades performance, confirming the necessity of both token-specific adaptivity and structured spatial support.

5. Significance and Claims

The paper claims that Token Radius Attention addresses a fundamental inefficiency in current sparse attention paradigms: the mismatch between shared allocation budgets and token-specific attention demands. By leveraging the intrinsic relationship between entropy and attention sparsity, TRA achieves a favorable quality-efficiency trade-off without requiring additional training or complex online ranking mechanisms.

The authors position TRA as a complementary technique to other acceleration methods (like caching or quantization) that focuses specifically on reducing the quadratic query–key interaction cost within executed attention layers. The work demonstrates that token-level adaptivity can be realized efficiently through structured, radius-based masking, offering a scalable path for deploying high-fidelity video generation models.