🤖 machine learning

QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling

QAM-W introduces a joint 2D codebook quantization method for LLM weights that utilizes Hadamard rotation and activation-aware scaling to preserve pairwise coordinate structures, achieving near-BF16 perplexity at approximately 5.5 bits per weight while outperforming polar coding and matching SmoothQuant quality with significantly fewer weight bits.

Original authors: Preetam Sharma, Kacper Dobek

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Preetam Sharma, Kacper Dobek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of books (a Large Language Model, or LLM) that is incredibly smart but takes up a huge amount of space. To make it easier to carry around, you want to shrink the books down. This process is called quantization.

Most current methods of shrinking these books are like taking a dictionary and replacing every single word with a short number code, one word at a time. It's simple, but it ignores the fact that words often come in pairs or groups that have a specific relationship to each other. If you treat them as isolated individuals, you lose some of the story's nuance.

This paper introduces a new method called QAM-W (Quadrature Amplitude Modulation for Weights). Think of it as a smarter, more efficient way to pack those books.

Here is how it works, using simple analogies:

1. The Problem: The "Solo Dancer" vs. The "Duet"

Imagine the weights in an AI model are dancers.

Old Method (Scalar Quantization): The old way treats every dancer as if they are performing a solo. It looks at each dancer individually and says, "You are a '3', you are a '7'." It ignores that two dancers might be holding hands or moving in sync.
The QAM-W Insight: The authors realized that within a row of data, these "dancers" are actually dancing in pairs. They are correlated. Instead of coding them as two separate soloists, QAM-W treats them as a duet.

2. The Solution: The "Magic Spin" and the "Pairing"

QAM-W uses a three-step process to shrink the model without losing the story:

Step A: The Magic Spin (Hadamard Rotation):
Imagine the dancers are standing in a messy, crowded room. QAM-W applies a "Magic Spin" (a mathematical rotation) that rearranges them. Suddenly, the dancers who were moving randomly are now paired up perfectly, moving in a smooth, circular pattern. This makes them much easier to describe mathematically.
Step B: The "Duet" Codebook:
Instead of giving each dancer a separate number, QAM-W looks at the pair as a single point on a map. It uses a pre-made "map" (a codebook) trained on how these pairs usually behave. It's like having a library of standard dance moves for duets. Instead of describing two separate steps, you just say, "They did the 'Waltz #42' move." This captures the relationship between the two numbers, saving space.
Step C: The "Volume Knob" (Activation-Aware Scaling):
Sometimes, certain parts of the model are very loud (highly active) and others are quiet. If you shrink the loud parts too much, the music gets distorted. QAM-W listens to the "volume" of each channel before shrinking. It turns down the volume of the loud channels slightly so they fit better into the small space, ensuring the most important information isn't crushed.

3. The Results: Smaller Size, Same Quality

The paper tested this new method on five different AI models (ranging from small to medium-large).

The "Sweet Spot": At a specific compression level (about 5.5 bits per weight), QAM-W achieved results that were almost identical to the original, uncompressed model.
The Comparison: A popular competitor method called SmoothQuant needed to use about 8.1 bits to get the same quality. QAM-W got the same result using 32% fewer bits.
- Analogy: It's like packing a suitcase. SmoothQuant needs a large suitcase to fit all your clothes neatly. QAM-W folds the clothes so efficiently that you can fit the exact same amount into a much smaller bag.

4. What It Doesn't Do (The Limits)

The paper is very specific about what it doesn't claim:

Not the Smallest: If you try to shrink the model even more (down to 4 bits), another method called QTIP actually performs better. QAM-W is the champion in the "medium-small" range (5–6 bits), not the "tiny" range.
Storage vs. Speed: The paper measures how much space the model takes up on a hard drive or in memory. It does not yet claim to make the model run faster on a computer chip, because the software to actually use these compressed numbers in real-time is still being built.

Summary

QAM-W is a new "packing technique" for AI models. By recognizing that data comes in pairs, spinning them into a better shape, and adjusting for volume, it can shrink AI models by about a third without making them any less smart. It's a specialized tool that works best in a specific size range, offering a significant saving in storage space compared to current methods.

Technical Summary: QAM-W

Problem Statement

Current post-training quantization methods for Large Language Models (LLMs) are predominantly scalar, mapping each weight independently to an integer index. While simple, scalar coding discards the pairwise coordinate structure inherent within weight rows. The authors argue that within a weight row, coordinates are typically correlated; after an orthogonal rotation, these coordinates approximate a 2D circular Gaussian distribution. Scalar codebooks fail to exploit this structure, leaving "rate-distortion slack" that could be recovered by quantizing pairs jointly as points in a plane rather than as independent scalars.

Existing approaches address parts of this problem: rotation-based methods (e.g., QuIP, QuIP#) homogenize coordinate magnitudes but often rely on scalar or lattice coding; activation-aware methods (e.g., AWQ) scale channels based on activation magnitude but leave the joint coordinate structure untouched. The paper posits that a codec combining rotation, joint 2D coding, and activation-aware scaling can achieve higher quality at lower bitrates.

Methodology: The QAM-W Codec

The authors introduce QAM-W (Quadrature Amplitude Modulation for Weights), a codec that processes weight matrices row-by-row through the following pipeline:

Separation of Scale and Direction: The row norm ( $r = \|w\|_2$ ) is recorded as metadata. The row is normalized to a unit vector $u$ .
Deterministic Rotation: A sign-masked block-Hadamard rotation is applied to the normalized row ( $y = R_{fwd} u$ ). This rotation is an exact isometry in real arithmetic, meaning it introduces no mathematical distortion but redistributes energy to reduce coordinate coherence.
Pairing and Calibration: Adjacent rotated coordinates are grouped into complex values ( $z_k = y_{2k} + i y_{2k+1}$ ). Each pair is scaled by a calibrated factor $\sigma_k$ , estimated empirically from the mean amplitude of the pairs, modeling them as circular Gaussian.
Joint 2D Quantization: Instead of quantizing amplitude and phase independently (polar coding), QAM-W maps each normalized pair to the nearest centroid in a single 2D Lloyd-Max codebook trained on the unit circular Gaussian distribution.
Activation-Aware Scaling: To align codec error with the layer's output sensitivity, the method applies per-channel scaling factors ( $s_j \propto r_j^\alpha$ ) before quantization, where $r_j$ is the RMS of the input channel activations. This attenuates the contribution of high-RMS channels, following the logic of AWQ.
Decoding: The decoder reverses the process: unpacking indices, looking up centroids, rescaling by $\sigma_k$ , applying the inverse rotation, and multiplying by the stored row norm.

The authors analyze the system through a composite bound chain: the rotation is an isometry; the joint 2D coding minimizes distortion better than polar coding at fixed rates; and the layer-output error is governed by a trace identity ( $\|X\Delta W^\top\|_F^2 = \text{Tr}(\Delta W \hat{M}_x \Delta W^\top)$ ), which motivates the activation-aware scaling.

Key Contributions

Codec Design: A row-norm-factored, block-Hadamard-rotated, joint 2D Lloyd-Max codec for LLM weights with activation-aware scaling.
Theoretical Analysis: A composite monotone bound linking codec distortion (Lloyd distortion $D_B$ ) to the Kullback-Leibler (KL) divergence between BF16 and quantized models. The bound suggests that realized KL should rank methods similarly to the codec distortion.
Cross-Model Empirical Study: Evaluation across five LLMs from four families (1.1B–13B parameters) and eight quantized configurations.
Rank Correlation Discovery: The paper demonstrates that across 37 (method, model) configurations, paired KL divergence and $\Delta$ PPL% are rank-correlated at Spearman $\rho = 0.99$ , validating the use of KL as a diagnostic for quantization quality.

Results

The evaluation focuses on three instruction-tuned models (TinyLlama-1.1B, Qwen2.5-3B, Mistral-7B) and a frontier comparison on Llama-2-7B.

Quality at ~5.5 bpw: The activation-aware variant of QAM-W (QAM-W-5.5) stays within ±0.4% of BF16 perplexity on all tested models. This matches the quality envelope of SmoothQuant W8A8 (which operates at ~8.1 bpw) but achieves it with approximately 32% fewer weight bits.
Joint 2D vs. Polar: At equal bitrate (~4 bpw), the joint 2D coding (QAM-W-4) outperforms the polar baseline (QAM-W-polar) by 1–3 percentage points (pp) in $\Delta$ PPL on quantization-tolerant models (TinyLlama, Mistral) and by 14.6 pp on the quantization-sensitive Qwen2.5.
Comparison with QTIP: At a strict 4 bpw, the QTIP method (using E8 lattice trellises) outperforms QAM-W. The authors explicitly state that QAM-W's contribution is the 5–6 bpw quality-preserving band, not the strict 4 bpw frontier.
Low-Bit Variant: A 3.5 bpw variant (QAM-W-3.5) is competitive on quantization-tolerant architectures (e.g., Mistral 7B, +1.98% $\Delta$ PPL) but underperforms on sensitive ones (Qwen 2.5, +7.5% $\Delta$ PPL) compared to GPTQ.
Downstream Tasks: The codec preserves performance on six downstream tasks (PIQA, HellaSwag, etc.) and MMLU, with QAM-W-5.5 matching BF16 within 0.2 pp on MMLU across all models.

Significance and Claims

The paper claims that QAM-W-5.5 represents a quality-preserving choice for LLM compression, offering a significant reduction in weight storage and memory bandwidth (32% fewer bits) compared to the current state-of-the-art SmoothQuant W8A8, without sacrificing perplexity or downstream task accuracy.

The authors are modest about the scope of their claims:

They do not claim superiority at strict 4 bpw, acknowledging QTIP's dominance in that regime.
They note that the current pipeline dequantizes weights to BF16 before matrix multiplication; thus, the reported bit-per-weight (bpw) savings apply to storage and memory bandwidth, not inference latency, until a fused kernel is implemented.
The study is limited to dense decoder-only transformers with standard attention; extensions to contemporary architectures (e.g., Gemma 2, Llama 3) or 70B+ parameter scales are left for future work.
The codebook is trained on a synthetic unit circular Gaussian rather than per-checkpoint empirical distributions, though the authors suggest a learned codebook could yield further gains.

In summary, QAM-W demonstrates that recovering the pairwise coordinate structure of weights via joint 2D coding, combined with rotation and activation-aware scaling, allows for high-fidelity quantization in the 5–6 bpw regime, bridging the gap between the efficiency of 4-bit methods and the quality of 8-bit methods.