🤖 machine learning

Statistically-Lossless Quantization of Large Language Models

This paper introduces Statistically-Lossless Quantization (SLQ), a layer-wise asymmetric quantization method that achieves both task-lossless accuracy below 4 bits per parameter and distribution-lossless fidelity at 5–6 bits, while delivering 1.7–3.7x inference speedups over FP16.

Original authors: Michael Helcig, Eldar Kurtic, Dan Alistarh

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael Helcig, Eldar Kurtic, Dan Alistarh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of artificial intelligence as a massive, bustling library where the books are not made of paper, but of pure math. These "books" are Large Language Models (LLMs), the super-smart computers that can write stories, solve equations, and chat like humans. But there's a catch: these libraries are enormous. To read a book, you need a giant, expensive, and power-hungry supercomputer. Most people don't have one of those sitting in their living room. So, scientists have been trying to shrink these books down—making them smaller and lighter so they can fit on regular phones or laptops. This process is called "quantization." Think of it like taking a high-definition photo and compressing it into a smaller file size. Usually, when you shrink a file too much, the picture gets blurry or pixelated (this is "lossy" compression). If you want the picture to stay perfect, the file stays huge (this is "lossless" compression). The big question scientists have been asking is: Can we shrink these AI brains so much that they fit on a phone, but still keep them smart enough to pass a test, without making them stutter or hallucinate?

This paper, titled "Statistically-Lossless Quantization of Large Language Models," dives right into that tricky middle ground. The authors, Michael Helcig, Eldar Kurtic, and Dan Alistarh, argue that we don't need to choose between a blurry, tiny model and a perfect, giant one. Instead, they propose a new way to shrink these models that is "statistically lossless." Here's the magic trick: they realized that when humans (or even the AI itself) answer questions, there's always a tiny bit of randomness. If you ask the same AI the same question twice, it might give slightly different answers, just like how you might answer a friend's question differently depending on your mood. The authors found that if the shrunken model stays within that natural "mood swing" range, it's good enough. They created a method called SLQ (Statistically-Lossless Quantization) that acts like a master tailor. Instead of cutting every part of the AI's brain to the same small size, it measures exactly how sensitive each part is. Some parts get a tiny bit of space (as low as 3.3 bits per parameter), while others get a bit more.

The paper makes a few very specific discoveries that change how we think about shrinking AI. First, they proved that a specific way of measuring the "shrinkage" is crucial. They introduced a metric called EAR (Expected Acceptance Rate), which is like a "token agreement score." If the original AI and the shrunken AI agree on 99 out of 100 words they would pick, that's a win. They found that to get this perfect agreement, you need to be careful about how you shrink the numbers. They showed mathematically that using a "symmetric" way of shrinking (which treats positive and negative numbers the same) actually creates more noise and errors than an "asymmetric" way (which shifts the scale to fit the data perfectly). It's like trying to fit a lopsided suitcase into a square box; if you force it into the center, it won't close right. You have to shift the suitcase to fit the shape of the box.

The results are quite impressive. Using their SLQ method, they managed to shrink models like Llama-3.3-70B and Qwen3 down to an average of 3.3 to 4.7 bits per parameter while keeping their "task-lossless" accuracy (meaning they still pass the same tests as the big version). If they wanted to be even stricter and ensure the AI's internal "mood" (its probability distribution) was almost identical to the original, they needed about 5 to 6 bits. But the best part? Because the models are so much smaller, they run much faster. The authors tested this on real hardware and found that their shrunken models were 1.7 to 3.7 times faster than the original, full-size versions, and they could even fit on fewer graphics cards. They also showed that older methods, like GPTQ or AWQ, often miss the mark on this "statistically lossless" goal, either being too blurry or not compressing enough. In short, this paper suggests that by being smarter about where and how we shrink the AI, we can make these giant brains small, fast, and still incredibly accurate, without needing to sacrifice the quality of their answers.

Technical Summary: Statistically-Lossless Quantization of Large Language Models

Problem Statement

Current Large Language Model (LLM) quantization approaches face a binary trade-off: lossy methods (e.g., GPTQ, AWQ) achieve high compression and inference acceleration but degrade model accuracy, while lossless methods (e.g., ZipNN, entropy-coded representations) preserve fidelity but often fail to provide inference acceleration or achieve high compression ratios.

This paper identifies a gap between these extremes: the regime of Statistically-Lossless Quantization (SLQ). The authors argue that since most open-weight LLMs are deployed with stochastic sampling parameters (e.g., temperature > 0), their outputs naturally exhibit run-to-run variance. A quantized model that stays within this natural sampling variance is, for practical purposes, lossless. The challenge is to define and achieve compression that preserves the model's output characteristics under standard sampling without requiring bit-exact reconstruction.

Methodology

The proposed SLQ framework utilizes layer-wise non-uniform scalar quantization with asymmetric quantization grids. The methodology is built on three core pillars:

1. Theoretical Foundation: Asymmetric vs. Symmetric Quantization

The paper proves that asymmetric quantization is a prerequisite for distribution-lossless fidelity, whereas symmetric quantization is insufficient for this strict goal.

Centering Inefficiency ( $\gamma$ ): Defined as $\gamma = 2M/R$ , where $M$ is the maximum absolute magnitude and $R$ is the dynamic range. For symmetric distributions, $\gamma=1$ ; for asymmetric distributions (typical in LLMs), $\gamma > 1$ .
Gamma-Squared Variance Law: The authors prove that symmetric quantization inflates output noise variance by a factor of $\gamma^2$ $γ^{2}$ relative to asymmetric quantization.
- $\sigma^2_{sym} = \gamma^2 \sigma^2_{asym}$
Impact: This variance inflation leads to higher probabilities of "decision flips" (changing the argmax token), particularly in low-entropy regions where the model is most confident. Consequently, symmetric quantization requires significantly more bits (up to 1.5 bits per parameter more) to achieve the same distributional fidelity as asymmetric quantization.

2. Fidelity Metrics

To navigate the near-lossless regime, the paper introduces two distinct definitions of losslessness and corresponding metrics:

Task-Lossless (TL): Preserves zero-shot benchmark accuracy within natural sampling variance.
- Metric: Recovery rate (percentage of BF16 accuracy retained).
- Search Strategy: Single-point calibration. Leveraging the observed linear relationship between KL divergence and task recovery in the near-lossless regime, the authors derive a KL threshold from a single benchmark measurement to predict bitwidth requirements without exhaustive search.
Distribution-Lossless (DL): Requires the quantized model's next-token distribution to be practically indistinguishable from the original.
- Metric: Expected Acceptance Rate (EAR). Inspired by speculative decoding, EAR measures the maximum token-agreement probability between the original and quantized distributions under optimal coupling.
- Definition: $EAR = \frac{1}{N} \sum_{i} \sum_{k=1}^{K} \min(p_i(k), q_i(k))$ , restricted to the top- $K$ tokens. An EAR of 0.99 implies 99% agreement under optimal coupling.
- Search Strategy: Direct optimization on EAR constraints using calibration data.

3. The SLQ Pipeline

Sensitivity Estimation: Uses Multi-Bitwidth Shapley Estimation to evaluate the marginal impact of assigning specific bitwidths (from a set $\{2, \dots, 8\}$ ) to different layer groups. This extends prior binary-choice methods to a full integer bitwidth range.
Bitwidth Allocation: Formulates the allocation as a Multiple-Choice Knapsack Problem solved via Integer Linear Programming (ILP). The solver minimizes predicted degradation subject to a bitwidth budget or fidelity constraint.
Implementation: The method uses asymmetric integer group quantization (group size 128) and is compatible with standard inference kernels (e.g., Humming kernels in vLLM).

Key Results

The authors evaluated SLQ on four models (Qwen3-8B, Qwen3-32B, Qwen3.5-27B, and Llama-3.3-70B-Instruct) using weight-only quantization.

Accuracy and Compression

Task-Lossless (TL): Achieved at aggressive bitwidths ranging from 3.3 to 4.7 bits per parameter (bpp).
- Example: Llama-3.3-70B achieved 99.1% recovery at 3.59 bpp, outperforming uniform 4-bit quantization.
- Example: Qwen3.5-27B achieved 99.1% recovery at 3.30 bpp.
Distribution-Lossless (DL): Achieved at 5.0 to 6.6 bpp with EAR $\ge 0.99$ $\geq 0.99$ .
- Example: Qwen3-32B reached 100.1% recovery at 4.98 bpp.
- Example: Llama-3.3-70B reached 99.9% recovery at 6.55 bpp.
Comparison: SLQ significantly outperforms state-of-the-art methods like QTIP and YAQA in the distribution-lossless regime. For instance, on Llama-3.1-8B, QTIP at 4 bits achieved an EAR of 0.9519, whereas SLQ achieved a much lower KL divergence (0.004) at 5.59 bpp.

Inference Speedup

Using optimized Humming kernels on NVIDIA L40s GPUs, SLQ configurations provided 1.7× to 3.7× throughput over BF16 baselines.

Hardware Efficiency: Reduced model size allowed larger models to fit on fewer GPUs (e.g., Llama-3.3-70B fit on 2 GPUs instead of 4).
Comparison: SLQ consistently outperformed FP8 baselines in throughput per GPU.

Ablation and Analysis

Asymmetry: Removing asymmetric quantization (forcing symmetry) increased the bitwidth required for DL targets by ~1.5 bpp.
Bitwidth Range: Restricting the search to binary choices (e.g., {4, 8}) resulted in higher average bitwidths compared to searching the full range {2–8}.
Token-Level Analysis: In DL configurations, 98.1% of tokens matched the baseline exactly; divergences were limited to minor stylistic variations (synonyms, punctuation) with no factual errors.
Thinking Tokens: DL configurations introduced no additional "thinking" token overhead compared to BF16, whereas lossy configurations (TL and Uniform-4) increased thinking tokens by 2.2–2.9%.

Significance and Claims

The paper claims to establish a new paradigm for LLM compression by formalizing statistically-lossless quantization. Its primary contributions are:

Formalizing the Gap: Distinguishing between task-level and distribution-level losslessness, arguing that the latter is necessary for true fidelity in stochastic decoding scenarios.
The $\gamma^2$ Law: Providing the first formal proof connecting asymmetric quantization design to token-level decision flips, demonstrating that symmetric quantization is fundamentally inefficient for distribution-lossless goals.
The EAR Metric: Introducing a directly interpretable fidelity metric (Expected Acceptance Rate) that complements KL divergence.
Practical Efficiency: Demonstrating that statistically-lossless compression is achievable at sub-4-bit (for task-lossless) and ~5–6 bit (for distribution-lossless) widths while delivering significant inference speedups, effectively bridging the gap between lossy compression and lossless storage.

The authors position SLQ not as a replacement for all quantization methods, but as a principled approach for the "near-lossless" regime where small distributional shifts are critical, offering a Pareto-dominant solution over uniform quantizers.