QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification

QuantSparse is a unified framework that effectively combines model quantization and attention sparsification for video diffusion transformers by introducing Multi-Scale Salient Attention Distillation and Second-Order Sparse Attention Reparameterization to mitigate information loss, thereby achieving significant storage reduction and inference acceleration while substantially outperforming existing baselines in generation quality.

The Gist

🎬 The Problem: The "Hollywood Blockbuster" That Won't Fit in Your Pocket

Imagine you have a Hollywood blockbuster movie (a high-quality AI video generator like HunyuanVideo or Wan2.1). This movie is amazing, but it's so huge and heavy that:

1. It needs a massive warehouse to store the film reels (it takes up 20GB+ of computer memory).
2. It takes a whole day to project a single scene (it takes nearly an hour to generate a video).

Because of this, you can't run these movies on a normal laptop or phone. They are too expensive and slow for real life.

Scientists have tried two tricks to make these movies smaller and faster:

• Trick A (Quantization): Compressing the film reel. Instead of using high-definition 4K color, they use a simpler, lower-quality palette (like turning a 16-bit color image into a 4-bit sketch). This saves space but can make the picture look grainy or weird.
• Trick B (Sparsification): Ignoring parts of the movie. They decide that 85% of the pixels are "boring" and just delete them, only keeping the important parts. This makes it fast, but if you delete too much, the movie looks like it's missing chunks.

The Catch: If you try to do both at the same time (compress the colors AND delete the pixels), the movie falls apart. The graininess from the compression makes the missing chunks look even worse. It's like trying to listen to a radio station that is both static-filled and missing half the broadcast.

🚀 The Solution: QuantSparse (The "Smart Editor")

The authors of this paper created a new tool called QuantSparse. Think of it as a super-smart film editor that knows exactly how to compress and cut the movie without ruining the story. It uses two special techniques to fix the problems mentioned above.

1. The "Map & Highlight" Trick (Multi-Scale Salient Attention Distillation)

The Problem: When you compress the movie (Trick A), the "map" of where the camera should look gets blurry. The AI gets confused about what is important.

The Solution: The editor uses a two-step guide to keep the AI on track:

• The Global Map (Low-Res): The editor first looks at a tiny, blurry thumbnail of the whole movie. This helps the AI understand the big picture (e.g., "This is a beach scene"). It's cheap to compute and keeps the structure right.
• The Highlighter (High-Res): The editor then zooms in on the most important parts of the movie (like a sea turtle swimming or a cliff edge). It tells the AI, "Ignore the boring water, but pay super close attention to the turtle."

The Analogy: Imagine you are trying to memorize a map of a city.

• Old Way: You try to memorize every single crack in the sidewalk (too much info) or just look at a blurry photo (not enough info).
• QuantSparse Way: You look at a small map to see the main roads (Global), and then you use a highlighter to mark the specific coffee shops you need to visit (Local). You get the best of both worlds without memorizing the whole city.

2. The "Time-Traveling Glue" (Second-Order Sparse Attention Reparameterization)

The Problem: Videos are made of frames that happen one after another. If you delete pixels (Sparsification), you lose some "glue" that holds the movement together. Usually, the AI tries to guess the missing glue by looking at the previous frame. But because the movie is also compressed (Quantization), the "glue" changes slightly every second, making the guess wrong.

The Solution: The editor realizes that while the "glue" changes, the way it changes is actually very stable.

• First-Order (The Old Way): "The glue was here yesterday, so it's probably here today." (Often wrong because of compression noise).
• Second-Order (The New Way): "The glue moved this specific direction yesterday, and the change in that movement is very predictable."

The Analogy: Imagine you are walking down a hallway.

• First-Order: You guess your next step based on where you are now. If the floor is slippery (compression noise), you might slip.
• Second-Order: You notice that even though the floor is slippery, your tendency to slip to the left is very consistent. So, you adjust your walk based on that consistent pattern. You aren't just guessing your position; you are predicting the change in your position, which is much more stable.

By using this "Time-Traveling Glue," QuantSparse can fill in the missing pieces of the video so accurately that the final result looks almost identical to the original, huge movie.

🏆 The Results: The Magic Numbers

When they tested this on the biggest video models (like HunyuanVideo and Wan2.1):

• Storage: They shrunk the model size by 3.8 times (like turning a 50GB hard drive into a 13GB one).
• Speed: They made the video generation 1.8 times faster.
• Quality: The video quality remained almost perfect. In fact, on some tests, it looked better than other compressed methods because it focused so well on the important details.

🎯 The Takeaway

QuantSparse is like a master chef who can take a giant, expensive, slow-to-cook feast and turn it into a quick, portable meal without losing any flavor.

• Before: You needed a supercomputer to watch AI videos.
• Now: With QuantSparse, we can run these high-quality video generators on much smaller, cheaper devices, making them ready for real-world use (like on your phone or in a web browser).

It solves the "impossible triangle" of AI: Fast + Small + High Quality. Usually, you can only pick two. QuantSparse proves you can have all three.

Technical Summary

1. Problem Statement

Video Diffusion Transformers (DiTs), such as Wan2.1-14B and HunyuanVideo-13B, have achieved state-of-the-art (SOTA) video generation capabilities but suffer from prohibitive computational and memory costs. Generating a single high-resolution clip can require over 20GB of GPU memory and nearly an hour of inference time, hindering real-world deployment.

While Model Quantization (reducing precision to low-bit integers) and Attention Sparsification (pruning redundant attention connections) are two promising compression techniques, they face a critical bottleneck when combined:

• Naive Integration Failure: Simply applying both techniques together leads to severe performance degradation.
• Amplified Attention Shift: The paper identifies that sparsification removes low-magnitude weights, while quantization introduces systematic perturbations (noise) to the remaining weights. These two effects reinforce each other, causing a compounded distortion in attention distributions. This "amplified attention shift" destroys the fine-grained dependency modeling required for high-quality video generation.

2. Methodology: QuantSparse Framework

The authors propose QuantSparse, a unified framework that synergistically integrates quantization and sparsification to overcome the amplified attention shift. The framework consists of two core components:

A. Multi-Scale Salient Attention Distillation (MSAD)

• Goal: To align the quantized and sparse attention maps with the original Full-Precision (FP) model during the Post-Training Quantization (PTQ) calibration phase, mitigating the bias introduced by compression.
• Challenge: Storing full attention matrices for large video models (sequence length $L > 10^4$) is memory-prohibitive ($O(L^2)$).
• Solution: A memory-efficient distillation scheme using two parallel guidance branches:
  1. Global Guidance: Downsamples Query ($Q$) and Key ($K$) tokens via average pooling to capture coarse structural topology. This provides low-resolution supervision at $O(\tilde{L}^2)$ cost.
  2. Local Guidance: Leverages the observation that video attention is highly skewed (heavy-tailed distribution). It identifies the top-$k$ "salient" tokens that dominate the attention mass and applies high-resolution supervision only to these critical tokens.
• Optimization: The quantization parameters are optimized to minimize a combined loss function ($L_{quant} + \lambda_{global}L_{global} + \lambda_{local}L_{local}$), ensuring the compressed model mimics the FP attention patterns without storing full matrices.

B. Second-Order Sparse Attention Reparameterization (SSAR)

• Goal: To recover information lost during inference due to sparsity and quantization noise, which standard caching methods fail to handle.
• Insight:
  • First-Order Residual: The difference between full and sparse attention ($\Delta(t) = A_{full} - A_{sparse}$) is unstable under quantization because quantization noise varies across timesteps.
  • Second-Order Residual: The difference between consecutive first-order residuals ($\hat{\Delta}(t) = \Delta(t) - \Delta(t-1)$) exhibits significantly higher temporal stability. Quantization noise follows a slow-varying stochastic process, making the second-order difference approximately stationary.
• Mechanism:
  1. Cache: During inference, the system caches the first-order residual and the second-order residual from a reference timestep.
  2. Reparameterization: The full attention output is approximated by adding the cached residuals to the sparse attention output: $\tilde{A}(t) = A_{sparse} + \Delta_{cached} + \hat{\Delta}_{cached}$.
  3. SVD Projection: To further reduce variance, the second-order residual is projected onto its top-$r$ principal components via Singular Value Decomposition (SVD), extracting the most temporally stable subspace. This adds negligible computational overhead while significantly improving accuracy.

3. Key Contributions

1. Theoretical Analysis: Formalized the "amplified attention shift" problem, proving that naive integration of quantization and sparsification causes compounded distortions that degrade video quality.
2. Unified Framework: Proposed QuantSparse, the first framework to seamlessly combine aggressive quantization (W4A8) and sparsification (15% density) without quality loss.
3. Novel Techniques:
   • MSAD: Introduced a memory-efficient, multi-scale distillation strategy that balances global structure and local salience.
   • SSAR: Developed a second-order reparameterization method exploiting temporal stability to recover lost information, outperforming first-order caching.
4. Comprehensive Evaluation: Validated the approach on large-scale models (1.3B to 14B parameters), demonstrating superior efficiency-quality trade-offs compared to existing baselines.

4. Experimental Results

The authors evaluated QuantSparse on HunyuanVideo-13B and Wan2.1-14B under aggressive settings (W4A8 quantization and 15% attention density).

• Performance Quality:

  • HunyuanVideo-13B: Achieved 20.88 PSNR and 81.19 VQA score. This surpasses the SOTA quantization-only baseline Q-VDiT (16.85 PSNR) and is nearly lossless compared to the Full-Precision model (81.23 VQA).
  • Wan2.1-14B: Achieved 18.22 PSNR and 90.73 VQA, maintaining near-perfect visual quality compared to the FP model (90.79 VQA).
  • Comparison: QuantSparse significantly outperforms naive combinations of quantization and sparsification (e.g., Q-VDiT + SVG), which suffer severe degradation.

• Efficiency Gains:

  • Storage: Reduced model storage by 3.68× (Hunyuan) and 3.80× (Wan2.1).
  • Memory: Reduced peak memory consumption by 1.32× to 1.51×.
  • Latency: Achieved 1.88× end-to-end inference acceleration (Hunyuan) and 1.74× (Wan2.1).

• Ablation Studies: Confirmed that removing MSAD or SSAR leads to significant performance drops, validating the necessity of both components. The method is also robust to hyperparameter choices (pooling stride, salient token count).

5. Significance

QuantSparse represents a breakthrough in making large-scale video generation models practical for deployment on resource-constrained hardware.

• Breaking the Trade-off: It successfully breaks the traditional trade-off between efficiency and performance, achieving "Pareto optimal" results where high compression does not sacrifice visual fidelity.
• Scalability: The method is applicable to models ranging from 1.3B to 14B parameters and generalizes to image generation tasks (tested on Hunyuan-DiT).
• Practical Impact: By reducing storage requirements by nearly 4× and inference time by nearly 2× while maintaining SOTA quality, QuantSparse enables the deployment of high-fidelity video generation models on consumer-grade GPUs or edge devices, facilitating broader adoption in real-world applications.