🤖 machine learning

InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

InfoQuant is a training-free post-training quantization method that employs Peak Suppression Orthogonal Transformation (PSOT) and adaptive outlier-token selection to reshape activation distributions into a compact, well-dispersed form, thereby significantly reducing quantization error and outperforming existing low-bit LLM deployment baselines.

Original authors: Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too-Tall" Box

Imagine you have a giant, chaotic pile of sand (this represents the data inside a Large Language Model, or LLM). You want to pack this sand into a small, neat box to save space and make it easier to carry (this is quantization—shrinking the model so it runs faster and uses less memory).

Usually, most of the sand is a normal, manageable height. But, there are a few giant spikes of sand sticking way up into the sky (these are called outliers).

Because of these few giant spikes, you are forced to use a huge box to fit everything in. Once you put the sand in that huge box, the "normal" sand at the bottom gets squished into a tiny, flat layer. When you try to take the sand out later, you can't tell the difference between the different layers of normal sand because they've all been flattened into the same spot. The model loses its ability to "think" clearly.

The Old Solutions: Flattening the Spikes

Previous methods tried to fix this by:

Smashing the spikes: Trying to cut the top off the giant sand piles.
Moving the spikes: Shifting the heavy sand from one spot to another.

The problem is that even if you cut the spikes, the remaining sand might still be clumped together in a way that doesn't fit well into the small box. You might have a smaller box, but the sand is still bunched up in a way that makes it hard to distinguish between different grains.

The New Idea: INFOQUANT

The authors of this paper, INFOQUANT, realized that the goal isn't just to make the sand "smaller." The goal is to make the sand look like it was designed for the box.

They ask a new question: "What does a pile of sand look like that fits perfectly into a small box?"

Their answer: It needs to be short (so it fits in the box) but also spread out evenly (so you can see every grain of sand).

How INFOQUANT Works (The 3-Step Recipe)

1. The "Spin and Spread" (Peak Suppression Orthogonal Transformation)
Imagine the sand is in a spinning top. The authors use a special mathematical trick (an orthogonal rotation) to spin the sand.

What it does: It takes those giant spikes and spreads their energy out across the whole pile.
The Result: The giant spikes disappear, and the sand becomes a smooth, wide hill. Crucially, the hill is now shorter (fits in the box) but wider (the sand is spread out so you can distinguish the layers).

2. The "Smart Scout" (Adaptive Outlier-Token Selection)
Sometimes, the sand pile has a few weird, noisy spots that look like spikes but aren't really important. If you try to fix those, you might mess up the good parts.

What it does: INFOQUANT has a "scout" that looks at the sand and says, "Okay, that spike there is real and dangerous, let's fix that. But that tiny bump over there? Ignore it."
The Result: The model focuses its energy on fixing the real problems, making the process more robust and less likely to get confused by noise.

3. The "Fine-Tuning Clip" (Learnable Activation Clipping)
After spinning and spreading the sand, the authors do one last check. They adjust the exact size of the box to make sure the sand fits perfectly without any gaps or crushing.

The Result: The final package is optimized for the specific size of the box, ensuring no information is lost in the corners.

Why This Matters

The paper tested this on famous AI models (like LLaMA-2 and LLaMA-3).

The Result: When they shrunk the models down to 4-bit (a very small size, like shrinking a 100-page book down to a postcard), INFOQUANT kept 97% of the original intelligence.
The Comparison: Previous methods lost much more intelligence when shrinking to this size. INFOQUANT managed to keep the "postcard" readable and useful, whereas others turned it into a blurry scribble.

The Bottom Line

Think of INFOQUANT as a master packer. Instead of just trying to force a messy, spiky pile of sand into a small box (which crushes the details), they first reshape the pile so it naturally fits the box perfectly. They make the pile short enough to fit, but spread out enough to keep all the details visible.

This allows us to run huge, powerful AI models on smaller, cheaper computers without losing their "brainpower."

Technical Summary: INFOQUANT

Problem Statement

Low-bit activation quantization remains a critical bottleneck for the efficient deployment of Large Language Models (LLMs). While weights in LLMs are typically flat and uniformly distributed, activations often contain a small number of dominant coordinates (outliers) that expand the quantization range. This forces round-to-nearest quantization to map many normal values to the same few discrete levels, resulting in significant information loss.

Existing Post-Training Quantization (PTQ) methods attempt to mitigate this by suppressing outliers, balancing channels, or minimizing reconstruction error (e.g., MSE). However, the paper argues that these approaches fail to explicitly define what activation distribution is actually "quantization-friendly." Methods may reduce visible peaks or lower numerical error without improving the fundamental "discretizability" of the distribution, often leaving the quantization range too wide or causing values to collapse near the mean.

Methodology: INFOQUANT

The authors propose INFOQUANT, a training-free PTQ framework that recasts activation transformation as "quantizer-facing distribution design." The core philosophy is derived from an information-theoretic analysis showing that low quantization error requires two complementary properties:

A smaller numerical range (to reduce the quantization step size).
Sufficient dispersion within that range (to utilize available quantization levels effectively).

INFOQUANT achieves this through three main components:

1. Peak Suppression Orthogonal Transformation (PSOT)

PSOT learns an orthogonal rotation matrix to reshape activation distributions. Unlike fixed rotations (e.g., Hadamard), PSOT optimizes a block-diagonal orthogonal matrix $R$ to minimize a specific loss function:
$\ell_{ps}(x; R) = \|\pi_T(y) \odot y\|^2$
where $y = xRP_\perp$ (centered rotated activation), and $\pi_T$ is a softmax weighting based on magnitude. This objective explicitly penalizes high-magnitude coordinates (peaks). By suppressing the largest centered coordinate while preserving the total token energy (due to orthogonality), PSOT forces the remaining energy to spread across more dimensions. This simultaneously reduces the clipping scale (narrowing the range) and increases the normalized dispersion ( $b_n$ ).

2. Adaptive Outlier-Token Selection (ASOT)

To ensure robust optimization, INFOQUANT does not treat all calibration tokens equally. It identifies tokens with "quantization-sensitive outlier structures" and reweights them during the rotation optimization.

It selects tokens containing at least one statistically extreme coordinate.
It employs a threshold selection mechanism to ensure the selected outlier positions vary across samples (avoiding overfitting to fixed prompt layouts or calibration artifacts).
These selected tokens are up-weighted ( $\gamma > 1$ ) in the loss function, while normal tokens serve as regularizers.

3. Learnable Activation Clipping (LAC)

After the distribution is reshaped by PSOT, the method applies a final calibration step using learnable clipping parameters ( $\alpha, \beta$ ). These parameters refine the activation step size to match the quantized block output to the full-precision output, minimizing the reconstruction error specifically for the reshaped distribution.

Key Contributions

Information-Theoretic Framework: The paper establishes a theoretical link between quantization error and activation distribution properties. It demonstrates that minimizing quantization error is equivalent to reducing the normalized clipped error bound, which requires a smaller numerical range and higher dispersion.
INFOQUANT Algorithm: A hardware-efficient, training-free method that learns orthogonal transformations to shape activations into quantization-friendly distributions, complemented by adaptive token selection and learnable clipping.
Empirical Validation: Extensive experiments across LLaMA-2 and LLaMA-3 families (7B to 70B) and Qwen2.5 models.

Experimental Results

The method was evaluated under various bit-width configurations, including the challenging W4A4KV4 setting (4-bit weights, 4-bit activations, 4-bit KV cache).

Accuracy Preservation: In the W4A4KV4 setting, INFOQUANT preserves an average of 97% of the original floating-point accuracy across LLaMA-2 and LLaMA-3 models.
State-of-the-Art Performance: It reduces the performance gap of the LLaMA-2 13B model by 42% compared to the previous state-of-the-art (SpinQuant) under W4A4KV4.
Comparison with Baselines: INFOQUANT consistently outperforms rotation-based methods (QuaRot, SpinQuant) and weight-reconstruction methods (GPTQ, AWQ). It also outperforms the distillation-based OSTQuant on several models (e.g., LLaMA-3 8B, LLaMA-2 13B) without requiring full-precision supervision.
Efficiency: The method enables the quantization of the LLaMA-2 70B model using only 24GB of GPU memory. It maintains a lightweight inference path compatible with existing low-bit kernels, avoiding the high FLOP overhead associated with dynamic transformations during inference (unlike FlatQuant).

Significance and Claims

The paper claims that the primary limitation of current low-bit activation quantization is not merely the presence of outliers, but the mismatch between the activation distribution and the capabilities of a low-bit uniform quantizer. By shifting the focus from "outlier suppression" to "distribution shaping," INFOQUANT demonstrates that learning specific orthogonal transformations can significantly improve discretizability.

The authors emphasize that their approach is training-free and hardware-efficient, making it practical for deployment on consumer-grade hardware. They note that while their method relies on calibration data and specific implementation choices (like block-diagonal matrices), it offers a robust alternative to supervised fine-tuning or heavy distillation pipelines. The paper concludes that future work should extend this analysis beyond bell-shaped activation assumptions to broader distributional regimes.