🤖 machine learning

KronQ: LLM Quantization via Kronecker-Factored Hessian

KronQ is a post-training quantization framework that improves large language model compression by incorporating gradient covariance into the quantization objective via Kronecker-factored Hessian approximation, enabling bidirectional incoherence processing and enhanced sensitivity metrics that achieve superior perplexity compared to existing methods like GPTQ, particularly in extreme low-bit scenarios.

Original authors: Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart robot brain (a Large Language Model) that can write stories, solve math problems, and chat like a human. The problem? This brain is so massive it needs a warehouse full of computers just to fit on a shelf. To make it run on a regular laptop or phone, scientists try to "shrink" it by compressing its memory, a process called quantization. Think of it like packing a suitcase: you want to fit all your clothes (the model's knowledge) into a tiny bag without losing anything important.

For a while, the best way to pack this suitcase was a method called GPTQ. It worked by looking at how the robot's "input" (the words it reads) varied and adjusting the weights accordingly. But the authors of this new paper, KronQ, noticed a flaw in this strategy. They realized that GPTQ was only looking at the input side of the equation, assuming that every possible output (the words the robot might say next) was equally important.

The Big Mistake: Ignoring the "Output" Side
The paper argues that this assumption is like trying to pack a suitcase by only looking at how heavy the clothes are, while ignoring how fragile they are. In reality, some parts of the robot's brain are super sensitive. If you squish them too much, the whole thing falls apart.

The authors measured this and found something wild: in a massive model like LLaMA-3-70B, the "output" channels (the paths the robot uses to generate answers) vary wildly in importance. Some are like delicate glass; others are like sturdy bricks. The old methods treated them all the same, which caused the robot to go haywire when compressed too much. In fact, when they tried to shrink LLaMA-3-70B down to just 2 bits (a tiny amount of memory), the old methods completely failed, producing nonsense with a "perplexity" (a score of how confused the model is) of over 2000. That's like a robot forgetting how to speak English entirely.

The New Solution: KronQ
Enter KronQ. The authors propose a new way to pack the suitcase that looks at both the input and the output. They use a mathematical trick called the Kronecker-factored Hessian approximation.

Here's the analogy: Imagine the robot's brain is a giant drum.

Old Method (GPTQ): You tap the drum from the front (input) and adjust the tension based on how the front vibrates. You assume the back of the drum is just a mirror image.
KronQ: You tap the drum from the front and the back. You realize the back vibrates differently! Some parts of the back are loose and wobbly, while others are tight. KronQ measures this "back vibration" (the gradient covariance) and uses it to pack the suitcase much more carefully.

Two Superpowers of KronQ
The paper shows KronQ does two clever things:

Bidirectional Incoherence (The "Shuffle"):
Before packing, KronQ shuffles the weights around in two directions (input and output) to make sure no single spot is overloaded with heavy or fragile items. It's like taking a messy pile of books and spreading them out evenly so the suitcase doesn't bulge in one spot. The paper shows this reduces the "variability" of the weights, making them much easier to compress without breaking.
Smart Bit Allocation (The "VIP Pass"):
Not all parts of the robot need the same amount of space. KronQ calculates a "sensitivity score" for every sub-layer of the brain. It gives the "VIP" layers (the most sensitive ones) more bits (more memory space) and the less important ones fewer bits. Crucially, because it looks at the output side, it can tell the difference between layers that look identical from the input side but act very differently.

The Results: Saving the Day
The authors tested this on several models, from 7 billion to 70 billion parameters. The results were dramatic, especially when they tried to shrink the models to the extreme limit of 2 bits.

The Failure: On the massive LLaMA-3-70B model, the old methods (GPTQ and GPTAQ) diverged or produced garbage results (perplexity > 2000).
The Success: KronQ managed to compress this same 70-billion-parameter model to 2 bits and kept it working perfectly, achieving a perplexity of 7.93 on the WikiText-2 dataset. That's a massive difference between "broken robot" and "smart robot."

Even on 4-bit and 3-bit settings, KronQ consistently beat the competition, showing lower confusion scores and better accuracy on reasoning tests like PiQA and Arc-Challenge. For example, on LLaMA-2-7B at 2 bits, KronQ scored a perplexity of 8.15, while GPTQ struggled at 31.11.

The Cost
Is there a catch? The paper admits that to get this "output vibration" data, KronQ needs to do one extra backward pass through the model before packing. This takes a little more time and memory during the setup phase (about 8–11 seconds more per layer compared to GPTAQ). However, once the model is packed, the actual running speed and memory usage during use are just as efficient as the older methods. The extra work is a one-time setup cost that pays off by allowing the model to be shrunk much further without breaking.

What They Ruled Out
The paper explicitly argues against the idea that you can ignore the output side of the math. They show that assuming all output directions are equal (setting the gradient covariance to an identity matrix) is a "suboptimal approximation" that leads to failure in ultra-low-bit scenarios. They also show that methods which only look at input statistics (like standard GPTQ) cannot distinguish between different sub-layers that share the same input, leading to poor decisions on where to allocate memory.

How Sure Are They?
The authors are very confident in these results because they are based on hard measurements, not just simulations. They ran these tests on real hardware (A100 GPUs) with real models (LLaMA-2, LLaMA-3, Gemma, DeepSeek) and measured actual perplexity and accuracy scores. They didn't just suggest it might work; they showed that in these specific experiments, the old methods literally failed to produce a working model at 2 bits, while KronQ succeeded.

In short, KronQ is like a master packer who realized that to fit a giant brain into a tiny box, you have to understand how the brain outputs information, not just how it inputs it. By doing so, it unlocks the ability to run massive AI models on devices that previously thought it was impossible.

Technical Summary: KronQ: LLM Quantization via Kronecker-Factored Hessian

Problem Statement

Post-training quantization (PTQ) is a critical technique for deploying Large Language Models (LLMs) on resource-constrained hardware without retraining. While existing second-order PTQ methods, such as GPTQ and its successors (e.g., GPTAQ), have become the standard for one-shot quantization, they rely on a fundamental limitation: they construct quantization objectives exclusively from input activation statistics ( $H_X$ ).

Under the Kronecker-factored approximation (K-FAC), the full weight Hessian ( $H$ ) factorizes into the Kronecker product of the input activation covariance ( $H_X$ ) and the output gradient covariance ( $H_G$ ). However, compensation-based methods implicitly assume $H_G = I$ (the identity matrix), treating all output directions as equally important. The authors argue this is a suboptimal approximation, as Figure 1(a) demonstrates that output channels exhibit heterogeneous gradient magnitudes. This oversight leads to performance degradation, particularly in ultra-low-bit regimes (e.g., 2-bit), where methods like GPTQ and GPTAQ often diverge or produce degenerate quantizations on models like LLaMA-3-70B.

Methodology

The paper proposes KronQ, a PTQ framework that explicitly incorporates the gradient covariance ( $H_G$ ) into the quantization pipeline under the K-FAC approximation ( $H \approx H_X \otimes H_G$ ). The methodology operates at two complementary levels:

1. Bidirectional Incoherence Processing (BiIP)

Standard incoherence processing (e.g., in QuIP#) applies random orthogonal transforms to suppress outliers in the input dimension ( $H_X$ ) to reduce quantization error. KronQ extends this to the output dimension using $H_G$ .

Mechanism: The method applies diagonal rescaling and orthogonal transforms to both the input ( $V$ ) and output ( $U$ ) dimensions. The rescaling factors are derived from the diagonal entries of $H_X$ and $H_G$ relative to weight norms.
Invariance: Crucially, while $H_G$ is used to determine the preprocessing transformations, it cancels algebraically during the column-wise Optimal Brain Surgeon (OBS) weight compensation updates (Proposition 1). This allows KronQ to retain the computational efficiency of the base GPTAQ solver while benefiting from the improved weight distribution.
Result: BiIP reduces the coefficient of variation (CV) of weight magnitudes in both input and output dimensions, leading to tighter quantization error bounds.

2. Inter-Layer Mixed-Precision Allocation

KronQ introduces a new sensitivity metric for allocating bit-widths across sublayers.

Metric: The sensitivity score is defined as the product of the traces of the two Hessian factors: $s_\ell = \text{tr}(H_G^{(\ell)}) \cdot \text{tr}(H_X^{(\ell)})$ .
Advantage: Existing methods using only $\text{tr}(H_X)$ cannot differentiate between sublayers that share identical input statistics (e.g., Query, Key, and Value projections in attention blocks). The KronQ score breaks this degeneracy because $H_G$ varies across these sublayers due to different downstream gradients. This enables more optimal bit-width allocation, assigning higher precision to sublayers with higher joint sensitivity.

Key Contributions

Kronecker-Factored Quantization Error: The authors extend the layer-wise quantization objective to incorporate $H_G$ under the K-FAC approximation. They demonstrate that $H_G$ can be utilized for bidirectional incoherence processing without increasing the computational cost of the quantization update loop, as it cancels out algebraically.
Joint Hessian Trace Sensitivity: A new metric for mixed-precision allocation is derived from the joint traces of $H_X$ and $H_G$ . This metric resolves the degeneracy found in activation-only metrics, allowing for distinct sensitivity rankings for Q, K, and V projections.
Empirical Validation: Extensive experiments show that KronQ achieves state-of-the-art results across LLaMA-2 and LLaMA-3 models (7B to 70B) at W2/W3/W4 bit-widths. The most significant gains are observed at 2-bit, where competing methods often fail.

Experimental Results

The paper evaluates KronQ on LLaMA-2 (7B, 13B, 70B) and LLaMA-3 (8B, 70B), as well as newer models like Gemma-3 and DeepSeek-R1.

Uniform Precision: On LLaMA-3-70B with 2-bit weight-only quantization, GPTQ and GPTAQ diverge (producing perplexity >2000 on WikiText-2), whereas KronQ achieves a perplexity of 7.93. Similar robustness is observed across 3-bit and 4-bit settings.
Group Quantization: In 2-bit group quantization (g=128), KronQ significantly outperforms GPTQ (e.g., 7.61 PPL vs. 274.0 PPL on LLaMA-2-7B).
Weight-and-Activation Quantization: Using the QuaRot framework for activations, KronQ consistently lowers perplexity compared to baselines (e.g., reducing PPL from 36.74 to 9.38 on LLaMA-2-7B at W2A4).
Mixed-Precision: The KronQ sensitivity score yields strictly better perplexity–bit-width tradeoffs compared to activation-only scores. For instance, on LLaMA-2-7B, upgrading the "down proj" sublayer first (per KronQ) yields lower perplexity than upgrading the "gate proj" first (per activation-only metrics) at the same average bit-width.
Generalization: The method generalizes to newer model families (Gemma-3, DeepSeek-R1, Phi-4) and harder benchmarks (LiveCodeBench, AIME-2024), often doubling the performance of baselines on generative reasoning tasks.

Significance and Claims

The paper claims that KronQ addresses a critical blind spot in current PTQ literature: the neglect of output-side gradient statistics. By leveraging the Kronecker-factored structure, KronQ provides a principled way to handle output-side heterogeneity without incurring the heavy computational costs associated with full Hessian estimation or iterative solvers used in related works like YAQA.

The authors emphasize that the method is particularly vital for ultra-low-bit quantization (2-bit), where the assumption of equal output importance ( $H_G = I$ ) leads to catastrophic failure in large models. The ability to achieve stable 2-bit quantization on 70B parameter models represents a significant step toward deploying LLMs on edge hardware.

Limitations: The authors acknowledge that KronQ requires a single backward pass over the calibration set to estimate $H_G$ , introducing a moderate offline latency overhead and temporary peak memory usage during the BiIP preprocessing step. However, once precomputed, $H_G$ is released, and the quantization loop remains as efficient as GPTAQ. Inference overhead is limited to the orthogonal transforms, which are identical to those in QuIP#.