💻 computer science

Q-ARVD: Quantizing Autoregressive Video Diffusion Models

This paper introduces Q-ARVD, a novel quantization framework designed to overcome the unique challenges of autoregressive video diffusion models—specifically unbalanced frame-wise sensitivity and heterogeneous weight outliers—through a final-quality aware frame-weighting mechanism and an outlier-aware adaptive dual-scale quantization strategy, thereby enabling efficient and accurate real-time video generation.

Original authors: Siao Tang, Xinyin Ma, Gongfan Fang, Xingyi Yang, Xinchao Wang

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siao Tang, Xinyin Ma, Gongfan Fang, Xingyi Yang, Xinchao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to draw a movie, frame by frame, like a flipbook. This is what Autoregressive Video Diffusion Models (ARVDs) do. Instead of drawing the whole movie at once, the robot draws one frame, looks at it, and then uses that drawing to create the next one, and so on. This is great for making movies in real-time, but it's very heavy and slow because the robot has to do a massive amount of math for every single frame.

To make this robot faster and able to run on smaller devices (like a phone), the researchers tried to "shrink" its brain using a technique called quantization. Think of quantization like compressing a high-resolution photo into a smaller file size. Usually, you just squeeze the whole picture down. But the authors found that for these video robots, simply squeezing everything down makes the movie look terrible.

They discovered two main reasons why the standard "squeeze" fails, and they built a new tool called Q-ARVD to fix it.

The Two Big Problems

1. The "Butterfly Effect" of Early Frames
In a normal video, if you make a small mistake in the middle, it might be okay. But in this "flipbook" robot, if you make a tiny mistake in the first frame, that mistake gets passed to the second frame, which passes a bigger mistake to the third, and so on. By the end of the movie, the first tiny error has exploded into a huge disaster.

The Analogy: Imagine a game of "Telephone." If the first person whispers the wrong word, everyone after them gets it wrong. The first person's whisper is the most important.
The Issue: Standard compression treats every frame equally. It shrinks the first frame just as much as the last frame. But the first frame needs to be kept super clear, while the last frame can afford to be a bit fuzzier.

2. The "Outlier" Channels
Inside the robot's brain, there are thousands of tiny pathways (channels) that carry information. Most of these pathways carry normal-sized signals. But a few of them carry massive signals (outliers).

The Analogy: Imagine a highway where 99% of cars are small sedans, but one lane is occupied by a giant semi-truck. If you try to fit all the cars into a small parking lot (low precision), the giant truck takes up so much space that the sedans get crushed or can't fit at all.
The Issue: Standard compression tries to fit the truck and the sedans into the same small space. The truck forces the whole system to use a huge space, making the sedans (the normal data) very inaccurate. Also, the paper found that sometimes the "truck" is in the first layer of the brain, and sometimes it's in the last. You can't use the same rule for every layer.

The Solution: Q-ARVD

The authors created Q-ARVD, a smarter way to shrink the robot's brain.

Fix #1: The "VIP Seat" Strategy (Final-Quality Guided Frame-Weighting)
Instead of treating all frames the same, Q-ARVD realizes that the early frames are the "VIPs."

How it works: During the training process, the system checks: "If I compress this specific frame, how much does it ruin the entire final movie?"
The Result: It gives the early frames a "VIP seat" in the compression process. It keeps them very precise (high quality) because they matter most. It allows the later frames to be compressed more aggressively because errors there don't ruin the whole movie as badly.

Fix #2: The "Special Parking Spot" Strategy (Outlier-Aware Adaptive Dual-Scale)
Instead of forcing the giant truck and the sedans into the same parking spot, Q-ARVD gives them separate spots.

How it works: The system automatically scans each layer of the brain to find the "trucks" (outlier channels).
- If it finds a truck, it puts the truck in a special, larger parking spot (a separate quantizer).
- The sedans (normal channels) get their own, tighter parking spot.
The Result: Because the sedans aren't fighting for space with the giant truck, they can be packed much more efficiently without getting crushed. The system does this automatically for every layer, because it knows some layers have trucks and some don't.

The Results

When they tested this new method:

Quality: The compressed videos looked almost exactly like the original, high-quality videos. Other methods made the videos look blurry or weird (like changing the color of the sky or the shape of a dog).
Speed & Size: By using this method, they made the model 1.97 times smaller (almost half the size) and 1.3 times faster. This means the robot can now run much faster on real devices.

In short, Q-ARVD is like a smart packing manager who knows that the first few items in a suitcase are fragile and need special care, and who knows how to handle the one giant, awkward item so it doesn't ruin the packing of everything else. This allows the video generation robot to run faster without losing its mind.

Technical Summary: Q-ARVD

Problem Statement

Autoregressive Video Diffusion Models (ARVDs) have emerged as a critical architecture for streaming, real-time interactive video generation and world modeling, overcoming the latency limitations of traditional bidirectional diffusion models. However, the substantial inference cost of ARVDs hinders their practical deployment on resource-constrained devices. While model quantization is a standard technique for improving efficiency, directly applying existing quantization schemes—optimized for bidirectional diffusion transformers or Large Language Models (LLMs)—to ARVDs yields suboptimal performance.

The authors identify two critical, unique challenges in quantizing ARVDs that existing methods fail to address:

Highly Unbalanced Frame-wise Quantization Sensitivity: Due to the autoregressive nature of ARVDs, where current frames depend on previously generated ones, quantization errors in early frames compound rapidly. Empirical analysis reveals that quantization sensitivity follows an exponential-like decay along the temporal axis; early frames disproportionately govern the final video quality, yet standard calibration treats all frames equally.
Prominent and Heterogeneous Outlier Patterns: Weight distributions in ARVDs exhibit significant channel-wise outliers (e.g., a small fraction of channels with magnitudes far exceeding the median). Unlike static outlier patterns in other models, these outliers are highly heterogeneous, varying significantly across different layer types (e.g., self-attention vs. cross-attention) and block depths. A static solution is insufficient as some layers are well-behaved while others suffer from severe outliers.

Methodology: Q-ARVD

To address these challenges, the authors propose Q-ARVD, a novel quantization framework tailored specifically for autoregressive video diffusion models. The framework consists of two primary innovations:

1. Final-Quality Guided Frame-Weighting

To tackle the unbalanced sensitivity across frames, Q-ARVD introduces a mechanism that weights the quantization objective based on the impact of each frame on the final video quality.

Sensitivity Quantification: The framework defines frame-wise sensitivity ( $\alpha_i$ ) by measuring the quality degradation of the entire generated video when only the $i$ -th frame is quantized while the rest remain in full precision. This is calculated as the distance (e.g., Mean Squared Error in latent space) between the original full-precision video and the video generated with the specific frame quantized.
Weighted Reconstruction: The sensitivity scores, which exhibit an exponential decay pattern, are used as loss-weighting coefficients during the quantization reconstruction process. This ensures that the calibration process prioritizes preserving precision in critical early frames rather than treating all frames uniformly.

2. Outlier-Aware Adaptive Dual-Scale Quantization

To handle heterogeneous outlier patterns without manual tuning, Q-ARVD employs an adaptive strategy that automatically detects and isolates outlier channels.

Automatic Detection: For each layer, the method computes the L2 norms of input channels and applies a Modified Z-score criterion based on the Median Absolute Deviation (MAD). A channel is flagged as an outlier if its magnitude exceeds a threshold derived from the MAD and a minimum magnitude constraint relative to the median. This prevents "false outliers" in smooth layers where MAD is small.
Dual-Scale Strategy: Once outliers are detected, the framework applies a dual-scale quantization. It separates the weight matrix into outlier channels and normal channels, applying independent quantizers to each. By isolating the outliers, the scaling factor for the normal channels is reduced, thereby minimizing the rounding error for the majority of the weights. This process is applied adaptively; layers without detected outliers remain standard.

Key Contributions

Identification of ARVD-Specific Challenges: The paper formally identifies and analyzes the unbalanced frame-wise quantization sensitivity and the heterogeneous nature of weight outliers in autoregressive video diffusion models, distinguishing them from bidirectional diffusion and LLMs.
Q-ARVD Framework: The authors propose the first quantization framework specifically designed for ARVDs, featuring:
- A final-quality guided frame-weighting mechanism to align quantization calibration with the actual error propagation dynamics of autoregressive generation.
- An outlier-aware adaptive dual-scale quantization strategy that automatically detects and isolates heterogeneous outliers, reducing quantization error without manual intervention.
Comprehensive Evaluation: The work provides extensive experiments on state-of-the-art open-source ARVDs (self-forcing and causal-forcing), demonstrating that Q-ARVD significantly outperforms existing diffusion quantization baselines (e.g., MinMax, AdaRound, SmoothQuant, PTQ4DiT, SVDQuant).

Experimental Results

The authors evaluated Q-ARVD on causal-forcing and self-forcing models using reference-based metrics (FVD-FP and LPIPS-FP) and reference-free VBench scores.

Performance: Q-ARVD consistently achieved the best FVD-FP and LPIPS-FP scores across various bitwidths (W8A8, W4A8, W4A6). The improvements were particularly pronounced in low-bit settings (W4A8 and W4A6) where outlier issues are more severe.
Visual Quality: Visual comparisons showed that baseline methods suffered from accumulated errors (e.g., MinMax) or semantic distortions (e.g., SVDQuant), whereas Q-ARVD maintained high video quality consistently across the temporal span.
Efficiency: Practical deployment of the INT8 (W8A8) model demonstrated a 1.30x speedup in latency and a 1.97x reduction in model size on an NVIDIA A6000 GPU.
Metric Analysis: The authors noted that standard VBench metrics often lack discriminative power for quantization evaluation, showing counterintuitive results (e.g., lower bitwidths sometimes scoring higher). They demonstrated that reference-based metrics (FVD-FP, LPIPS-FP) are more sensitive and faithful to actual quality degradation.

Significance and Claims

The paper claims that Q-ARVD is the first quantization framework tailored specifically for autoregressive video diffusion models. Its significance lies in bridging the gap between the theoretical potential of ARVDs for real-time applications and the practical constraints of deployment. By addressing the unique error accumulation dynamics of autoregressive generation and the heterogeneous nature of model weights, Q-ARVD enables near-lossless visual quality at significantly reduced computational and memory costs. The authors suggest that their approach to frame-wise sensitivity-aware quantization and adaptive outlier detection offers a new path for optimizing next-generation generative video models.