💻 computer science

SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation

SQuad is a sub-quadratic attention distillation framework that compresses pretrained Video Diffusion Transformers to achieve $\mathcal{O}(n\sqrt{n})$ complexity, delivering quadratic-level video generation quality with significantly reduced computational costs and faster inference speeds.

Original authors: Animesh Karnewar, Denis Korzhenkov, Amirhossein Habibian, Mohsen Ghafoorian

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Animesh Karnewar, Denis Korzhenkov, Amirhossein Habibian, Mohsen Ghafoorian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Creating moving pictures with artificial intelligence has become one of the most exciting frontiers in modern computing. These systems, known as video diffusion models, work by starting with a chaotic cloud of static noise and gradually refining it into a coherent scene, frame by frame, based on a text description. To understand how they do this, imagine a vast grid of tiny digital pixels, where each pixel is a token carrying information about color, shape, and time. The engine that drives these models is a mechanism called self-attention. This mechanism allows every single token in the video to look at every other token to decide how they should relate to one another. It is this constant, all-seeing connection that gives the video its lifelike quality and ensures that a character's movement in the first second matches their appearance in the last. However, this power comes with a steep price. As the video grows longer or more detailed, the number of connections the system must calculate explodes, making the process incredibly slow and expensive, often limiting creators to just a few seconds of footage.

Researchers at Qualcomm AI Research have developed a new approach called SQuad to solve this problem without sacrificing the quality of the video. Instead of trying to simplify the complex connections or replace the core mechanism with a cheaper, weaker alternative, they found a way to reorganize how the system looks at the data. In the standard method, every token must check in with every other token, a process that becomes unmanageable as the video size increases. The team realized that in video generation, most of these connections are actually very weak; only a small, critical group of tokens truly needs to pay close attention to each other at any given moment. Building on this observation, they designed a two-step process. First, the system groups nearby tokens into small windows and lets them mix information locally. Then, it takes a second pass to let these windows communicate with each other across the entire video. This structure allows the model to maintain a full view of the scene while drastically cutting down the number of calculations required.

The results of applying this method to a large video generation model called Wan 2.2 are striking. The researchers took a powerful, pre-trained model and taught it to use this new, streamlined attention method through a process of distillation, where the new system learns to mimic the behavior of the original. The outcome is a model that produces videos of nearly identical visual quality to the original, scoring almost the same on rigorous quality tests, but with a massive reduction in cost. While the original model required 100 steps to generate a video, the new version achieves similar results in just six steps. In terms of speed, the time it takes to process a single step of the video generation dropped from roughly 47 milliseconds to just 4 milliseconds. The amount of computing power needed for these steps fell by a factor of nearly 67.

This efficiency gain is not just a theoretical improvement; it translates directly into what a user experiences. When tested on a standard high-definition video generation task, the new method generated a video in about half the total time of the original system, even when accounting for the entire generation pipeline. The researchers compared their method against other attempts to speed up video generation, some of which rely on complex, hybrid architectures that add millions of extra parameters to the model. In contrast, the SQuad approach requires no additional memory or specialized hardware, fitting neatly into existing systems. User studies confirmed that people could not reliably tell the difference between videos made by the original, heavy model and those made by the new, efficient version. In fact, in a head-to-head comparison, a significant portion of human viewers actually preferred the videos generated by the new method.

The significance of this work lies in its balance. Previous attempts to make these models faster often involved replacing the powerful attention mechanism with simpler, linear approximations, which frequently resulted in a noticeable drop in video quality. The SQuad method avoids this trap by keeping the core mathematical operation intact but changing the order in which it is applied. It proves that the full, quadratic complexity of checking every token against every other token is not strictly necessary to achieve high-fidelity results. By carefully structuring the flow of information, the researchers have shown that it is possible to generate long, high-resolution videos with a fraction of the computational cost. This opens the door for generating longer, more complex scenes on standard hardware, moving the field closer to the goal of creating unlimited, high-quality video content without the current bottlenecks of time and energy.

Technical Summary: SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation

Problem Statement

Video generation using Diffusion Transformers (DiTs) is currently bottlenecked by the computational cost of the Softmax Self-Attention mechanism. In standard DiTs, attention complexity scales quadratically, $O(n^2)$ , with the number of latent tokens $n$ . For video generation, where $n$ represents a spatio-temporal volume (often reaching tens of thousands of tokens), this quadratic term dominates both runtime and memory, limiting the achievable resolution, duration, and throughput.

Existing solutions attempt to mitigate this by replacing Softmax Self-Attention with linear $O(n)$ or low-rank $O(nk)$ approximations, or by using sparse masks and hybrid architectures. However, these approaches often suffer from a "stubborn quality gap" because they sacrifice the non-linearity and sharp, input-dependent selectivity of the full Softmax. Furthermore, hybrid designs that interleave expensive quadratic blocks with efficient ones introduce architectural complexity and often retain a dependence on the very quadratic operations they aim to reduce.

Methodology: SQuad

The authors propose SQuad (Sub-Quadratic Attention Distillation), a framework that reduces attention complexity to $O(n\sqrt{n})$ while retaining the genuine Softmax operation throughout. Instead of training a new Video DiT from scratch, SQuad distills a pretrained full-softmax DiT (specifically the Wan 2.2 5B and Wan 2.1 1.3B models) into a modified architecture.

1. SQuad-Attention Operator

The core innovation is a two-stage attention mechanism that factorizes the full $O(n^2)$ attention into a Local Pass and a Global Pass:

Local Pass ( $Attn_L$ ): Tokens mix within fixed windows of size $w$ .
Global Pass ( $Attn_G$ ): Tokens at the same relative position within each window mix across all windows.
Composition: The output of the local pass is fed into the value stream of the global pass. This nesting allows the global pass to operate on tokens that have already been locally mixed, recovering a full receptive field.
Complexity: By setting the window size $w = \sqrt{n}$ , the method achieves $O(n\sqrt{n})$ complexity. Crucially, both passes utilize the standard Softmax, preserving the expressivity lost in linear approximations.

2. Distillation Strategy

Since replacing the attention operator changes the function computed by the network, the pretrained weights are no longer optimal. The authors employ a two-stage distillation process:

Flow-Matching Supervised Fine-Tuning (SFT): The network is fine-tuned on video data (VIPE 1M dataset) to re-seat the weights under the new SQuad-Attention architecture.
Distribution Matching Distillation (DMD2): An improved step-distillation method is applied to recover the generation quality lost during the architectural change and to reduce the number of Neural Functional Evaluations (NFEs) required for sampling.

Key Contributions

Sub-Quadratic Complexity with Full Softmax: SQuad achieves $O(n\sqrt{n})$ complexity without abandoning the Softmax operation, addressing the quality gap inherent in linear or low-rank approximations.
Distillation Framework: The paper demonstrates that a pretrained quadratic DiT can be effectively fitted into a sub-quadratic architecture via a two-stage distillation process (SFT + DMD2), avoiding the prohibitive cost of training from scratch.
Hardware Efficiency: The method is designed to be hardware-friendly, requiring no specialized fused kernels or on-device optimizations, relying instead on standard tensor reshaping and standard attention blocks.

Experimental Results

The method was evaluated on the Wan 2.2 5B and Wan 2.1 1.3B text-to-video models.

Quality Preservation: On Wan 2.2 5B, SQuad achieves a VBench Total score of 83.20, effectively matching the original full-softmax baseline (83.08). In user preference studies, 35% of users preferred SQuad generations over the baseline, while 31% found them indistinguishable.
Efficiency Gains:
- Latency: SQuad reduces per-block attention latency by approximately 11× (from 47.10ms to 4.27ms) and end-to-end DiT latency by 2× (from 667ms to 314ms in compiled mode).
- Compute: Attention TFLOPs are reduced by ~67× (from 4.205 to 0.063).
- Sampling Speed: The distilled model generates high-quality videos in 6 NFEs, compared to the default 100 NFEs for the original model.
Comparison: SQuad outperforms other efficient attention methods (such as VSA, Jenga, Radial Attention, and Attention Surgery) in terms of latency and TFLOPs while maintaining comparable or superior quality, without requiring additional learnable parameters (unlike methods that add 72M–283M weights).

Significance and Claims

The paper claims that SQuad validates the hypothesis that the full $O(n^2)$ complexity of token-to-token communication may not be necessary for high-fidelity video generation. By proposing a "middle ground" that retains the Softmax's expressivity through a structured, sub-quadratic factorization, SQuad offers a practical path to generating longer, higher-resolution videos on current hardware.

The authors position SQuad not only as a distillation target for existing models but also as a candidate attention mechanism for pretraining large models from scratch. They acknowledge that their current strategy couples attention distillation with step distillation (DMD2) and suggest that future work could isolate the attention change from sampling speedups or explore composing more than two passes to further trade depth for efficiency.