6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models

1. Problem Statement

Video Diffusion Transformers (Video DiTs) have achieved state-of-the-art results in video generation but suffer from prohibitive memory usage and computational costs, hindering their deployment on consumer hardware.

• Limitations of Existing Quantization: Current Post-Training Quantization (PTQ) methods typically use static bit-width allocation (e.g., fixed INT8 or INT4 for all layers).
• The Core Issue: The paper observes that the sensitivity of activations to quantization is highly dynamic across denoising timesteps. A layer that is stable at one timestep may be highly volatile at another. Static policies either cause severe visual artifacts (flickering, hallucinations) in sensitive steps or fail to maximize compression in stable steps.
• Hardware Gap: New GPU architectures (e.g., NVIDIA Blackwell) support NVFP4 (a micro-scaling block format) but have removed support for INT4, creating a need for methods that leverage this new hardware-native format.

2. Methodology

The authors propose 6Bit-Diffusion, a unified, training-free framework that combines three key components to achieve aggressive acceleration while preserving quality:

A. Dynamic Mixed-Precision Quantization (DMPQ)

Instead of static allocation, DMPQ dynamically assigns precision (NVFP4 or INT8) to activation channels based on real-time temporal stability.

• Key Insight: There is a strong linear correlation between a Transformer block's input-output relative difference at timestep $t-1$ and the quantization sensitivity of its internal linear layers at timestep $t$.
  • High Variation ($\Gamma_{t-1}$): Indicates a volatile layer requiring high precision (INT8).
  • Low Variation: Indicates a stable layer that can safely use ultra-low precision (NVFP4).
• Mechanism:
  1. A lightweight predictor calculates the relative L1 distance ($\Gamma$) between the previous block's input and output.
  2. Using a pre-fitted linear model ($E_{rel} = \alpha \cdot \Gamma + \beta$), the system predicts the quantization error.
  3. Routing: If the predicted error exceeds a threshold, the layer is routed to INT8; otherwise, it is routed to NVFP4.
• Outlier Smoothing: To handle activation outliers that break low-bit quantization, the method employs an online Block Hadamard Transform (Fast Hadamard Transform) applied to local activation blocks, redistributing outliers without heavy overhead.

B. Temporal Delta Cache (TDC)

This component exploits the high temporal redundancy in video generation to skip computations entirely.

• Observation: The residual deltas ($\Delta_t = X_{out} - X_{in}$) of Transformer blocks are highly consistent across adjacent timesteps.
• Mechanism:
  • The system monitors the similarity between the last two computed deltas ($\Delta_{t-1}$ and $\Delta_{t-2}$).
  • If the similarity is high (indicating a smooth ODE trajectory), the current block computation is skipped.
  • The output is approximated by reusing the cached delta from the last full computation step.
• Error Control: An accumulated error metric tracks the potential drift from continuous skipping. If the error exceeds a threshold, the cache is refreshed, and the block is recomputed.

C. Purified Delta Refresh (PDR)

A critical innovation to prevent the combination of DMPQ and TDC from causing quality degradation.

• Problem: Naively combining quantization and caching leads to error accumulation. Quantization noise in a cached delta is linearly amplified every time the cache is reused.
• Solution: Before a delta is written to the cache, the system checks for "outliers."
  • If the input activation has a high outlier ratio, the layer is forced to use Full Precision (FP16/BF16) for that specific step to ensure the cached delta is "pure."
  • If the block resumes computation after a skip (where the dynamic routing metric is missing), it defaults to INT8 as a safe fallback.

3. Key Contributions

1. First Dynamic Mixed-Precision Framework for Video DiTs: Introduces a method that dynamically switches between NVFP4 and INT8 based on temporal sensitivity, addressing the limitations of static quantization.
2. Temporal Delta Cache (TDC): A caching mechanism that skips redundant block computations by leveraging the temporal consistency of residual deltas, synergizing with quantization.
3. Purified Delta Refresh (PDR): A novel mechanism that isolates quantization noise from the caching process, preventing the accumulation of errors that typically ruins video quality when combining these techniques.
4. Hardware-Native Optimization: Specifically designed for modern GPUs (NVIDIA Blackwell) by utilizing NVFP4, a format previously unsupported in video DiT quantization literature.

4. Experimental Results

The framework was evaluated on CogVideoX (2B and 5B parameter models) using the VBench and EvalCrafter benchmarks.

• Performance Metrics:
  • Speedup: Achieved 1.92× end-to-end inference acceleration compared to the full-precision (FP16) baseline.
  • Memory Reduction: Achieved 3.32× reduction in GPU memory usage.
  • Quality: Maintained visual fidelity comparable to the full-precision baseline.
    • On CogVideoX-2B, the method achieved an Aesthetic Quality score of 0.5437 (W4A6), outperforming static baselines like SmoothQuant (0.5332) and ViDiT-Q (0.5332).
    • It significantly reduced temporal flickering and geometric distortions seen in other quantization methods.
• Ablation Studies:
  • Uniform W4A4 quantization caused severe quality drops (Flow Score dropped from 5.84 to 4.25).
  • Combining DMPQ and TDC without PDR caused error drift (Flow Score dropped to 4.72).
  • The full framework (DMPQ + TDC + PDR) restored the Flow Score to 5.54, demonstrating the necessity of the purification step.

5. Significance

This work represents a significant step toward the practical deployment of high-fidelity video generation models on consumer-grade hardware. By moving beyond static quantization policies and integrating caching with error-aware quantization, the authors demonstrate that it is possible to achieve near-lossless compression (effectively ~6-bit average precision) without the severe artifacts that have historically plagued low-bit video generation. The method sets a new baseline for efficient inference in Video DiTs, making high-quality video generation more accessible and scalable.