💻 computer science

HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models

The paper introduces HQ-DM, a novel Quantization-Aware Training framework that utilizes a Single Hadamard Transformation to effectively mitigate activation outliers and prevent weight outlier amplification, thereby enabling high-performance low-bit quantization (W4A4 and W4A3) for diffusion models with significant improvements in image generation quality over existing state-of-the-art methods.

Original authors: Shizhuo Mao, Hongtao Zou, Qihu Xie, Song Chen, Yi Kang

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shizhuo Mao, Hongtao Zou, Qihu Xie, Song Chen, Yi Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a digital artist to paint a masterpiece, but instead of giving them a full studio with high-definition brushes, you hand them a tiny, low-resolution sketchbook. This is the challenge facing "diffusion models," a type of artificial intelligence that creates stunning images by slowly turning random static noise into clear pictures, step by step. Think of it like a sculptor who starts with a giant block of marble and chips away tiny pieces over and over until a statue emerges. The problem is that this process is incredibly heavy; it requires massive computers and a lot of memory, making it hard to run on everyday devices like phones or laptops.

To fix this, scientists use a trick called "quantization." Imagine taking a high-definition photo and shrinking it down to a pixelated version that takes up less space. In the world of AI, this means converting the model's super-precise numbers (which use a lot of memory) into smaller, simpler numbers (using fewer bits). However, there's a catch: when you shrink these numbers too much, the AI gets confused by "outliers." These are rare, extreme values in the data that act like a single, blindingly bright star in a dark sky. If you try to fit that star into a small, low-resolution grid, it distorts the whole picture, causing the AI to produce blurry or weird images. The big question is: how do we shrink the model without losing the details that make the art look good?

This is where a new study called HQ-DM steps in with a clever solution. The researchers found that the "outliers" causing the trouble in diffusion models are like stubborn guests at a party who are standing in the doorway, blocking everyone else. Previous methods tried to squeeze these guests in or move the furniture around, but it often made the room messier. The HQ-DM team discovered a better way: they use a mathematical tool called a Single Hadamard Transformation. You can think of this as a magical "shuffling" technique. Instead of trying to force the bright star into a small box, they spin the entire sky around so that the star's light is spread out evenly across the whole canvas. Suddenly, no single point is too bright to handle, and the AI can easily shrink the numbers without losing the image's quality.

What makes this approach special is how it handles the "weights" (the model's memory) versus the "activations" (the data moving through the model). Older methods tried to shuffle both, but this often created new bright spots in the memory, making things worse. HQ-DM is smarter: it only shuffles the moving data (activations) right before it gets shrunk, leaving the memory untouched. This is like rearranging the guests in the hallway without moving the furniture in the living room. Because of this, the method works perfectly with standard computer chips that are designed for fast, simple math, allowing the AI to run much faster.

The results of this "shuffling" trick are impressive. When the researchers tested their method on a popular image generator called LDM-4 using the ImageNet dataset (a collection of 256×256 pixel images), they found that their model could be shrunk down to very small sizes without losing its artistic touch. Specifically, when they used a very aggressive setting where both the memory and the data were shrunk to just 4 bits (W4A4), their method improved the image quality score (Inception Score) by 12.8% compared to the best previous method. Even more dramatically, when they pushed the data down to just 3 bits (W4A3)—a level where most other methods fail completely—their model improved the score by a staggering 467.73%.

The paper also shows that this method is efficient. It doesn't take much longer to train the model, and because it plays nicely with standard computer hardware, it can run about 1.10 to 1.14 times faster than previous attempts at similar low-bit quantization. The researchers tested this on different types of models, including those based on "Transformers" (a different AI architecture), and found that the "shuffling" trick worked just as well there. In short, HQ-DM suggests that by simply rearranging the data before shrinking it, we can make powerful AI art generators small enough to run on the devices we use every day, without sacrificing the beauty of the images they create.

Technical Summary: HQ-DM – Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models

1. Problem Statement

Diffusion models have established themselves as state-of-the-art generative frameworks for image synthesis; however, their deployment in real-time or resource-constrained environments is hindered by high computational costs and memory consumption. While model quantization offers a solution to reduce storage and accelerate inference, applying it to diffusion models presents unique challenges distinct from traditional classification or generative tasks.

The primary bottleneck lies in the non-stationary nature of activation distributions during the multi-step denoising process. Unlike static tasks, the scale, feature structures, and statistical properties of activations in diffusion models vary significantly across timesteps. Furthermore, these models exhibit long-tailed distributions with prominent outliers. Under low-bit quantization (e.g., 4-bit or lower), these outliers cause severe clipping and rounding errors. Crucially, these errors accumulate and amplify over the iterative sampling steps, leading to substantial degradation in generation quality. Existing Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ) methods struggle to mitigate these activation outliers effectively, often resulting in poor performance at low bit-widths.

2. Methodology: HQ-DM

The authors propose HQ-DM, a novel Quantization-Aware Training (QAT) framework designed to mitigate activation outliers while maintaining hardware efficiency. The core innovation is the application of a Single Hadamard Transformation to activation matrices prior to quantization.

2.1 Single vs. Double Hadamard Transformation

Previous works (e.g., QuaRot, Xi et al.) utilized a "Double Hadamard Transformation," applying the transformation to both weights and activations. The authors identify two critical limitations in this conventional approach for diffusion models:

Incompatibility with Convolution: The distributive property required for Double Hadamard does not apply straightforwardly to convolution operations, especially with multi-stride convolutions common in diffusion models.
Weight Outlier Amplification: Applying Hadamard transformation to weight matrices that originally have low outliers can inadvertently generate new outliers in the transformed weights, amplifying quantization errors.

HQ-DM introduces the Single Hadamard Transformation, which applies the transformation only to the activation matrices before quantization, leaving the weights untransformed.

Mechanism: The input activation matrix $X$ is multiplied by a block-diagonal Hadamard matrix $H$ (constructed from standard Hadamard matrices $H_k$ ). The transformed activations $(XH)$ are then quantized.
Recovery: Due to the orthogonality of the Hadamard matrix ( $H^T = H^{-1} = H$ ), the original values can be recovered during inference via the operation $\hat{X} = S_{XH} \cdot (XH)_{int} \cdot H$ .
Hardware Efficiency: By extracting the normalization factor $2^{-k/2}$ into the floating-point quantization scales, the core computation involves only integer matrix multiplication and convolution. This ensures compatibility with hardware accelerators supporting INT operations.

2.2 LoRA-Based Distillation Framework

HQ-DM integrates the Single Hadamard Transformation into a distillation framework inspired by EfficientDM:

Teacher-Student Setup: A full-precision (FP32) teacher model guides a quantized student model.
LoRA Adaptation: The original pre-trained weights are frozen. Instead, Low-Rank Adaptation (LoRA) matrices ( $A$ and $B$ ) are trained to adapt the model to the quantization noise. The adapted weight is $W' = W + B \cdot A$ .
Timestep-Aware Quantization: Recognizing the non-stationary nature of activations, the framework employs timestep-aware learnable quantization scales (LSQ) for both weights and activations, optimizing them independently for each denoising step.
Symmetric Quantization: Unlike some existing methods that use asymmetric quantization for activations, HQ-DM adopts symmetric quantization to ensure full compatibility with integer convolution operations.

3. Key Contributions

Single Hadamard Transformation Framework: The authors propose a distillation framework that applies Single Hadamard Transformation to activations to reduce outliers. This approach significantly enhances diffusion model performance under low-bit quantization settings.
Implementation-Friendly Design: The proposed Single Hadamard Transformation addresses the limitations of the Double Hadamard approach by:
- Enabling seamless support for integer (INT) convolution operations, which are critical for diffusion models.
- Preventing the amplification of weight outliers, thereby avoiding the introduction of additional quantization errors in the weight matrices.
Superior Low-Bit Performance: The method demonstrates exceptional performance across multiple datasets (ImageNet, LSUN) and architectures (U-Net based LDMs and Transformer-based DiTs), particularly in scenarios with ultra-low-bit activation quantization (e.g., W4A3).

4. Experimental Results

The authors evaluated HQ-DM on the ImageNet 256×256 dataset using the LDM-4 model, as well as on LSUN-Bedrooms, LSUN-Churches, and DiT-XL models.

ImageNet 256×256 (LDM-4):
- W4A4 Quantization: HQ-DM improved the Inception Score (IS) by 12.8% compared to the state-of-the-art EfficientDM.
- W4A3 Quantization: HQ-DM achieved a massive 467.73% improvement in IS over EfficientDM. It also showed improvements exceeding 100% across all four metrics (IS, FID, sFID, Precision).
LSUN Datasets: HQ-DM consistently outperformed EfficientDM and other baselines (PTQD, QuEST) across various bit-widths (W8A4, W4A4, W4A3, W4A2). For instance, on LSUN-Bedrooms, HQ-DM reduced FID by 5.56, 3.66, and 4.92 for W8A4, W4A4, and W4A3 settings, respectively.
DiT-XL: On Transformer-based models, HQ-DM achieved a relative IS improvement of 60.92% and a precision improvement of 13.79% over EfficientDM under W4A4 quantization.
Efficiency: Training and calibration times for HQ-DM are comparable to EfficientDM, indicating that the computational overhead of the sparse Hadamard transformation is negligible. Inference latency is reduced by approximately 10%–14% compared to Double Hadamard approaches due to better hardware compatibility.

5. Significance and Claims

The paper claims that HQ-DM provides a systematic solution to the key bottlenecks hindering the accurate quantization of diffusion models: non-stationary activation distributions and outlier accumulation. By introducing the Single Hadamard Transformation, the authors demonstrate that it is possible to achieve high-quality generation at ultra-low bit-widths (W4A3) without sacrificing hardware efficiency.

The authors emphasize that their method is particularly significant because it resolves the trade-off between outlier suppression and hardware compatibility. Unlike previous methods that either failed to support convolutions or degraded weight quality, HQ-DM supports integer convolution operations directly while effectively smoothing activation outliers. The results suggest that HQ-DM is a practical and generalizable solution for deploying efficient, low-bit diffusion models in real-world scenarios.