🤖 machine learning

Theory-optimal Quantization Based on Flatness

This paper introduces Bidirectional Diagonal Quantization (BDQ), a novel post-training quantization framework that derives a theory-optimal solution based on a new "Flatness" metric to effectively disperse activation outliers, thereby achieving state-of-the-art accuracy in low-bit Large Language Model quantization.

Original authors: Xiusheng Huang, Zhe Li, Xuanwu Yin, Lu Wang, Yequan Wang, Dong Li, Emad Barsoum, Kang Liu

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiusheng Huang, Zhe Li, Xuanwu Yin, Lu Wang, Yequan Wang, Dong Li, Emad Barsoum, Kang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Loud Neighbor" in a Quiet Room

Imagine you have a massive, incredibly detailed library of books (a Large Language Model, or LLM). To fit this library into a small backpack (your phone or a cheap server), you need to shrink the books down. This process is called quantization.

Usually, the books are written in high-definition ink (32-bit or 16-bit precision). To save space, you want to rewrite them using only a few simple colors (4-bit or even 2-bit).

The Issue: Most of the text in these books is normal, but every now and then, there is a sentence written in giant, neon-red letters that screams louder than everything else. In the paper, these are called outliers.

When you try to shrink the whole book to fit into a small space, the "giant neon letters" force the entire system to stretch out to accommodate them. This squishes all the normal text into a tiny, unreadable corner. The result? The book becomes garbled nonsense.

The Old Solutions: Trying to Rotate the Room

Previous methods tried to fix this by rotating the room or rearranging the furniture. They would spin the data around (using linear transformations) hoping the giant neon letters would blend in with the normal text.

The authors of this paper looked at these methods and said, "It's not working well enough." Even after spinning the room, the neon letters are still there, just in a different spot. They are still hogging all the space, leaving the rest of the data cramped.

The New Idea: "Flatness" and the Equalizer

The authors came up with a new way to think about the problem. Instead of just trying to hide the neon letters, they wanted to flatten the landscape.

Imagine the data as a hilly terrain. Most of the land is flat, but there are a few massive mountains (the outliers).

The Goal: You want the terrain to be as flat as possible (like a calm lake). This is what they call Flatness.
The Metric: They invented a math tool to measure how "bumpy" the data is. If the mountains are too high, the "Flatness" score is bad. If the land is smooth, the score is good.

They proved mathematically that the best way to flatten this terrain isn't by spinning it, but by using a two-sided stretching tool.

The Solution: BDQ (Bidirectional Diagonal Quantization)

The authors propose a new method called BDQ. Here is how it works, using a metaphor:

Imagine the data is a giant grid of people standing in rows and columns.

The Problem: A few people in specific rows and columns are giants (outliers).
The Old Way: You try to rotate the whole grid. The giants are still giants; they just face a different direction.
The BDQ Way: You give every single person in a specific row a pair of stretchy pants (a diagonal matrix) and every person in a specific column a pair of stretchy shoes (another diagonal matrix).
- If a row has a giant, you shrink the pants for everyone in that row.
- If a column has a giant, you shrink the shoes for everyone in that column.
- The Result: The giants get shrunk down to a normal size, and the tiny people get stretched up. Suddenly, everyone is roughly the same height. The "terrain" is flat.

Because the giants are no longer dominating the space, you can now compress the whole grid into a tiny backpack without losing any important details.

The "Overfitting" Trap: The Student Who Crammed

There was one more problem. When teaching the computer how to use these stretchy pants and shoes, they only showed it a tiny sample of data (like 128 sentences).

The computer was like a student who memorized the answers to those 128 specific practice questions perfectly but failed the real exam because it didn't understand the general rules. In technical terms, this is called overfitting.

To fix this, the authors added a special rule called Recursive Cross-Entropy Loss.

Analogy: Instead of just telling the student "This is the right answer," they also say, "Look at what you predicted was right, and make sure your confidence matches the actual answer."
This forces the computer to learn the general pattern of how to flatten the data, rather than just memorizing the specific practice sentences.

The Results: Packing a Suitcase Perfectly

The paper tested this new method on some of the smartest AI models in the world (like LLaMA-3 and DeepSeek).

The Test: They tried to shrink the models down to extremely small sizes (using only 4 bits for weights and 4 bits for activations, or even 2 bits for weights).
The Outcome:
- Previous methods made the AI "dumb" when shrunk this small (accuracy dropped significantly).
- BDQ kept the AI almost as smart as the original full-size version.
- In one extreme test (shrinking a 70-billion-parameter model to 2-bit weights), BDQ closed the performance gap by nearly 40% compared to the best existing methods.

Summary

The paper claims that by mathematically proving that "flatness" is the key to good compression, and by using a two-sided stretching tool (BDQ) combined with a smarter learning rule (RCE), they can shrink massive AI models down to tiny sizes without them losing their intelligence. They turned a bumpy, mountainous landscape into a smooth, flat plain that fits easily into a small backpack.

Technical Summary: Theory-optimal Quantization Based on Flatness

1. Problem Statement

Post-training quantization (PTQ) has become a standard technique for compressing and accelerating Large Language Models (LLMs). However, a primary challenge remains the presence of activation outliers—values with magnitudes significantly larger than the statistical expectation of the rest of the data. These outliers force the quantization scale to expand to accommodate them, causing the majority of normal data to be compressed into a narrow range of integer levels. This results in significant quantization errors and performance degradation, particularly at low bit-widths (e.g., W4A4 or W2A4).

While recent approaches attempt to mitigate outliers via linear transformations (such as rotation), the authors observe that these methods often fail to fully disperse outlier patterns, leaving concentrated magnitude distributions that still degrade performance. Furthermore, existing methods rely on heuristic strategies without establishing a direct mathematical relationship between outliers and quantization errors, nor do they optimize the distribution of the entire quantization space.

2. Methodology

The paper proposes Bidirectional Diagonal Quantization (BDQ), a framework grounded in a theoretical analysis of "Flatness." The methodology consists of three main components:

2.1 Mathematical Modeling of Outliers and Error

The authors first derive the mathematical relationship between outliers and quantization error. They demonstrate that when an outlier $|w_{outlier}|$ is significantly larger than the quantization range, the total quantization error $E[\epsilon^2]$ is dominated by the outlier and exhibits a quadratic relationship with the outlier's magnitude. Specifically, the error scales with $w_{outlier}^2$ , indicating that outliers are the key factor limiting quantization accuracy.

2.2 The Flatness Metric and Theoretical Optimal Solution

To quantify the distribution of outliers, the authors introduce a new metric called Flatness, inspired by information entropy. Flatness evaluates the uniformity of data distribution across a matrix by treating normalized squared elements as a probability distribution.

Definition: Flatness ( $F$ ) is defined as the sum of $p_{ij} \ln p_{ij}$ , where $p_{ij} = \frac{W_{ij}^2}{\alpha_i \beta_j}$ . Here, $\alpha_i$ and $\beta_j$ are learnable energy weight factors for rows and columns, respectively.
Optimization: The goal is to minimize $F$ (maximize entropy) subject to energy constraints. Through Lagrange multipliers, the authors prove that the unique optimal solution for achieving maximum Flatness is a transformation of the form $V = d_1 W d_2$ , where $d_1$ and $d_2$ are diagonal matrices. This structure allows for independent scaling of rows and columns to disperse outliers.

2.3 The BDQ Framework

Building on the theoretical optimal solution, BDQ implements the following:

Bidirectional Diagonal Transformations: For each fully connected layer in the LLM, BDQ learns two diagonal matrices ( $\Lambda_1, \Lambda_2$ ) to perform row-wise and column-wise scaling. This redistributes outlier magnitudes across both dimensions.
Hadamard Orthogonal Transformation: To further disperse outliers and break strong column correlations that diagonal scaling alone might not resolve, BDQ incorporates a learnable rotation matrix $R$ (constructed from a Hadamard matrix and an orthogonal matrix). The forward pass is reformulated as $y = Q(\Lambda_1 x \Lambda_2 R) \cdot Q(R^T \Lambda_2^{-1} W \Lambda_1^{-1})$ , ensuring equivalent transformation at the global network level.
Recursive Cross-Entropy (RCE) Loss: To address the issue of overfitting on small calibration sets (e.g., 128 samples), which is common in PTQ and hinders outlier mitigation, BDQ introduces a Recursive Cross-Entropy loss. This loss function simultaneously fits the label distribution and the model's predicted token distribution, preventing the model from falling into local optima and improving generalization.

3. Key Contributions

Theoretical Modeling: The paper is the first to model the mathematical relationship between outliers and quantization errors, proving that errors scale quadratically with outlier magnitude.
Flatness Metric: The authors propose the Flatness metric to quantify outlier distribution and mathematically derive the optimal solution (bidirectional diagonal scaling) for minimizing this metric.
BDQ Framework: They propose Bidirectional Diagonal Quantization, which utilizes learned diagonal operations and Hadamard rotations to achieve the theoretical optimal solution for Flatness.
Overfitting Mitigation: The introduction of Recursive Cross-Entropy loss addresses the overfitting problem inherent in small calibration sets, a limitation observed in prior state-of-the-art methods.

4. Experimental Results

Extensive experiments were conducted on LLaMA-2, LLaMA-3, and DeepSeek-R1-Distill models across various bit-width settings (W4A4, W2A4, W3A3).

W4A4 Setting: On LLaMA-3-8B, BDQ achieves less than a 1% accuracy drop compared to full-precision models, outperforming existing methods like QuaRot, SpinQuant, and FlatQuant.
W2A4 Setting (Challenging): In the extremely low-bit W2A4KV16 setting on DeepSeek-R1-Distill-LLaMA-70B, BDQ reduces the performance gap compared to the full-precision model by 39.1% relative to the latest state-of-the-art approaches.
Efficiency: The method demonstrates significant inference speedups (up to 3.44x on NVIDIA A100) and memory savings (up to 3.74x on AMD MI250) compared to full-precision models.
Ablation Studies: Experiments confirm that the combination of BDQ and RCE loss yields superior results compared to using BDQ with standard cross-entropy, validating the effectiveness of the proposed loss function in preventing overfitting.

5. Significance and Claims

The paper claims that BDQ establishes a new benchmark for post-training quantization. Its primary significance lies in moving beyond heuristic outlier mitigation to a theory-optimal approach derived from first principles. By mathematically proving that bidirectional diagonal scaling is the optimal solution for maximizing Flatness (and thus minimizing quantization error), the authors provide a rigorous foundation for future quantization methods.

The authors acknowledge limitations, noting that experiments were constrained by computing resources to models up to 70B parameters and specific GPU architectures. However, they assert that the theoretical framework and the demonstrated performance on large models (LLaMA-3-70B, DeepSeek-R1-Distill-70B) validate the method's potential for efficient, low-cost LLM deployment.