💬 NLP

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

PolarQuant is a post-training quantization method for large language models that achieves near-lossless compression by applying block-wise normalization and Walsh-Hadamard rotation to transform weights into an approximately Gaussian distribution, enabling highly accurate quantization without calibration data while serving as an effective preprocessing step for downstream INT4 quantizers.

Original authors: Caio Vicentino

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Caio Vicentino

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model like Qwen3.5). To fit this library into a small backpack (your laptop or phone), you need to shrink the books. This process is called quantization.

The problem with current shrinking methods is like trying to pack a suitcase by just throwing everything in randomly. You end up wasting space on huge, rare items (outliers) while crushing the common, important items (the bulk of the data) because there's no room left.

PolarQuant is a new, clever way to pack that suitcase. It doesn't just shrink the books; it rearranges them first so they fit perfectly.

Here is how it works, using simple analogies:

1. The Problem: The "Outlier" Problem

Imagine you are packing a suitcase. Most of your clothes are standard t-shirts and jeans. But you also have one giant, fluffy winter coat and a few tiny, delicate earrings.

Old Method (Absmax): You measure the size of the biggest item (the winter coat) and use that as the ruler for everything.
- Result: The winter coat takes up 90% of the suitcase space. Your t-shirts and jeans get squished into tiny, unrecognizable crumbs because the ruler is too big for them. The suitcase is full, but the clothes are ruined.

2. The Solution: The "Magic Shuffle" (Hadamard Rotation)

PolarQuant introduces a magical step before packing. It takes your messy pile of clothes (the model weights) and performs a Hadamard Rotation.

Think of this as a magic shuffle.

You take that giant winter coat and the tiny earrings.
The magic shuffle breaks the coat down into threads and mixes them with the threads of the jeans and shirts.
Suddenly, you don't have one giant item and many tiny ones. Instead, you have 128 piles of clothes that are all roughly the same size.

Why is this cool?
The paper's "Aha!" moment is that after this shuffle, the data looks like a perfect Bell Curve (a Gaussian distribution). In statistics, this is the "Goldilocks" zone where everything is predictable and evenly spread out.

3. The Packing: Perfect Fitting

Now that all your "clothes" (data points) are roughly the same size and shape, you can pack them efficiently.

The Centroids: Instead of guessing where to put things, PolarQuant uses a pre-calculated map (Lloyd–Max centroids) that tells you exactly where to place each item to minimize damage.
The Result: You can fit the entire library into a much smaller space with almost no loss of quality.

The "Secret Sauce": The Shuffle Does 98% of the Work

The authors did a test to see what was actually doing the heavy lifting. They found something surprising:

The Magic Shuffle (Hadamard Rotation): Accounts for 98% of the improvement.
The Perfect Map (Centroids): Only accounts for 2%.

It turns out that simply rearranging the data so it's uniform is the hardest part. Once you do that, even a simple packing method works wonders.

Why This Matters for You

Run AI on Your Laptop: A 9-billion parameter model usually needs a massive, expensive server GPU (18GB of memory). With PolarQuant, you can run it on a standard laptop or even an Apple Mac mini with just 4.8GB of memory.
No Training Needed: You don't need to re-teach the AI how to speak. It's a "post-training" method, meaning you can take an existing model and shrink it instantly without needing extra data.
Speed: It doesn't slow down the AI. In fact, because the data is packed so neatly, the computer can read it just as fast as before.

The "Double Packing" Trick

The paper also shows a cool trick: You can use PolarQuant to shrink the model to 5 bits, unpack it slightly, and then shrink it again to 4 bits using standard tools.

Analogy: Imagine compressing a file with ZIP, then opening it, and compressing it again with RAR. Usually, this ruins the file. But because PolarQuant "cleaned up" the data first, the second compression step works better than if you had just compressed the messy original file directly.

Summary

PolarQuant is like a professional packer who knows that if you just shuffle the items around so they are all the same size, you can fit a whole wardrobe into a carry-on bag without crushing a single shirt. It makes running powerful AI on everyday devices possible, fast, and free of charge.

1. Problem Statement

Deploying Large Language Models (LLMs) on consumer and edge hardware is hindered by memory constraints. A 9-billion parameter model in FP16 requires ~18 GB of VRAM, exceeding the capacity of most consumer GPUs. While 4-bit quantization reduces this to ~5–6 GB, standard methods like absmax quantization are suboptimal for LLM weights.

The Core Issue: LLM weights follow a near-Gaussian distribution, but absmax quantization assumes a uniform distribution over $[-\alpha, \alpha]$ . This mismatch wastes quantization levels on rare outlier magnitudes (tails) and concentrates errors in the high-density central region, leading to significant perplexity degradation.
Existing Limitations: Advanced methods like GPTQ and AWQ require calibration data and are computationally expensive. Other approaches (e.g., QuIP#, SpinQuant) often require modifying the model graph or learning rotation matrices, complicating deployment.

2. Methodology: PolarQuant

PolarQuant is a post-training weight quantization method that transforms weight blocks into a distribution amenable to optimal quantization without calibration data. It operates in four stages:

A. Block Decomposition and Normalization

The weight tensor is flattened and partitioned into blocks of size $d$ (typically $d=128$ ). For each block $b_i$ :

Compute the $\ell_2$ norm $r_i = \|b_i\|_2$ .
Normalize the block to the unit hypersphere: $\hat{b}_i = b_i / r_i$ .

B. Hadamard Rotation

A normalized Walsh–Hadamard matrix $H_d$ is applied to the normalized block:
$\tilde{b}_i = H_d \hat{b}_i$
Theoretical Insight: According to the Central Limit Theorem for projections on a hypersphere, if a vector is uniformly distributed on the unit sphere, its coordinates after an orthogonal rotation (like Hadamard) converge to a standard normal distribution $N(0, 1/d)$ . The Hadamard rotation "homogenizes" the coordinates, effectively transforming the heavy-tailed, non-uniform weight distribution into an approximately i.i.d. Gaussian distribution.

C. Scaling and Quantization

Scaling: The rotated coordinates are scaled by $\sqrt{d}$ to achieve a standard normal distribution $N(0, 1)$ .
Quantization: The values are quantized to the nearest centroid. The paper utilizes Lloyd–Max centroids, which are analytically derived to minimize Mean Squared Error (MSE) for a Gaussian source.
- Note: The paper finds that for 5-bit quantization, the rotation alone provides 98% of the gain, making the specific Lloyd–Max centroids less critical than the rotation itself.

D. Storage and Dequantization

Storage: Stores 8-bit quantized codes, 16-bit per-block norms, and a global centroid table.
Dequantization: The process is the exact inverse. Since the Hadamard matrix is self-inverse ( $H_d = H_d^{-1}$ $H_{d} = H_{d}^{- 1}$ ), dequantization involves:
1. Lookup centroids.
2. Scale by $1/\sqrt{d}$ .
3. Apply $H_d$ again.
4. Multiply by the stored norm $r_i$ .
- Overhead: Dequantization adds ~8 seconds to model load time but incurs zero runtime overhead during inference.

3. Key Contributions

Near-Lossless Compression: PolarQuant achieves a perplexity of 6.39 on Qwen3.5-9B (Q5 quantization), which is only +0.02 away from the FP16 baseline (6.37), without using any calibration data.
The Primacy of Rotation: An ablation study reveals that the Hadamard rotation alone accounts for 98% of the quality improvement. It reduces perplexity from 6.90 (absmax Q5) to 6.40. The Lloyd–Max centroids provide only marginal additional gains at 5-bit precision.
Effective Preprocessing for INT4: PolarQuant serves as a powerful "denoising" preprocessing step. When PolarQuant Q5 weights are dequantized and re-quantized by torchao INT4, the resulting perplexity (6.56) is significantly better than direct torchao absmax INT4 (6.68). This is because the rotation removes outliers, allowing the downstream group-wise absmax quantizer to utilize its dynamic range more efficiently.
Simplicity and Portability: The algorithm requires no gradient computation, no iterative optimization, and no model graph surgery. It is compatible with any downstream quantizer (torchao, GGUF, MLX) and runs efficiently on both NVIDIA GPUs and Apple Silicon.

4. Experimental Results

Experiments were conducted on Qwen3.5-9B (9B parameters) using WikiText-2 perplexity as the metric.

Method	Perplexity (PPL)	$\Delta$ vs FP16	VRAM	Throughput (tok/s)
FP16 Baseline	6.37	0.00	17.9 GB	45.7
Absmax Q5	6.90	+0.53	~18 GB	N/A
PolarQuant Q5 (Dequant)	6.39	+0.02	18.1 GB	45.9
PolarQuant Q5 + AWQ	6.43	+0.06	17.9 GB	45.8
PolarQuant Q5 + torchao INT4	6.56	+0.19	6.5 GB	43.1
Direct `torchao` INT4	6.68	+0.31	6.3 GB	43.3

Edge Performance: On an Apple Mac mini M4 (16 GB RAM), PolarQuant achieved 19.7 tok/s at 4.8 GB memory usage with a perplexity of 6.90.
Ablation: Removing the Hadamard rotation caused a massive drop in performance (PPL 6.90), confirming that distributional transformation is the key mechanism, not just the choice of centroids.

5. Significance and Conclusion

PolarQuant fundamentally shifts the paradigm of weight quantization by exploiting the Gaussian structure of rotated weight blocks rather than fighting against the non-uniformity of raw weights.

Theoretical Impact: It demonstrates that a simple, deterministic orthogonal rotation (Hadamard) can transform complex weight distributions into a form where standard optimal quantizers (Lloyd–Max) or even simple uniform quantizers perform near-optimally.
Practical Impact: It enables near-lossless 5-bit compression without calibration data and significantly boosts the quality of 4-bit inference when used as a preprocessing step.
Deployment: The method is highly efficient, adding negligible runtime cost and enabling high-fidelity LLM inference on consumer-grade hardware (e.g., running 9B models on Mac mini M4).

The paper concludes that Hadamard rotation is the essential component of this success, simplifying the path to efficient, high-quality LLM compression.