🤖 machine learning

RDQ: Residual Distribution Quantization for Large Language Models

This paper introduces RDQ, a post-training quantization framework that mitigates the super-linear accumulation of quantization noise in large language models by employing Cascaded Error Compensation to fit per-channel scales against drifted residual distributions, thereby achieving state-of-the-art perplexity at 3-bit and 4-bit precision with zero inference overhead.

Original authors: Prateek Singh

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prateek Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a secret message down a long, winding tunnel made of 32 connected rooms. In the original, high-quality version of this tunnel (called FP16), the message travels perfectly clear from start to finish. But when we try to shrink the message to fit into a tiny, compressed format (quantization) to save space on your phone or edge device, things start to go wrong.

For a long time, scientists thought the problem was just that each room was a little bit messy on its own. They thought, "If we just clean up Room 1, then Room 2, then Room 3, everything will be fine." But this new paper, RDQ (Residual Distribution Quantization), discovered that this "clean each room separately" approach is actually the wrong way to think about it.

The Real Culprit: The "Snowball" Effect

The authors found that the real problem is a snowball effect. Every time a message passes through a room, a tiny bit of "noise" or static gets added to it. In a normal tunnel, this doesn't matter. But in a compressed tunnel, that tiny bit of noise doesn't disappear; it gets carried over to the next room, where it mixes with new noise from that room.

By the time the message reaches the 32nd room, that tiny static has turned into a massive roar. The authors measured this "noise accumulation" (which they call residual stream drift) and found something amazing: the amount of noise perfectly predicts how bad the message sounds. In fact, they found a correlation so strong (Pearson r=0.999) that if you know how much the signal has drifted, you can predict exactly how confused the computer will get.

The Big Discovery: The Tunnel Isn't Smooth

Before this paper, most experts assumed the noise in these tunnels looked like a smooth, predictable hill (a "Gaussian" distribution). They built their cleaning tools based on that smooth hill.

But the authors went inside and took a look. They discovered that 84% of the rooms in the LLaMA-3-8B model don't have smooth hills at all! Instead, the noise is jagged, weird, and sometimes has two peaks (bimodal). It's like expecting a calm lake and finding a choppy ocean with sudden waves. Because the noise is so weird, the old "smooth hill" cleaning tools were failing miserably, especially when they tried to shrink the message down to just 3 bits (a very tiny size).

The Solution: "Cascaded Error Compensation" (CEC)

So, how do we fix a tunnel where the noise changes as you go deeper? The authors propose a new method called Cascaded Error Compensation (CEC).

Instead of cleaning each room while pretending the message is still fresh and clean (which is what old methods did), CEC says: "Let's walk through the tunnel first, see exactly what the message looks like when it gets to each room, and then clean it."

Here is how it works:

The Walk-Through: The computer runs a test message through the tunnel. It lets the first few rooms get "dirty" (quantized) just like they would be in real life.
The Real Look: When the message arrives at Room 10, it is already a bit noisy from Rooms 1 through 9. The system captures this actual, messy signal.
The Custom Fix: Instead of using a generic cleaning tool, the system builds a custom cleaning tool specifically for that messy signal. It adjusts the "volume" (scales) of the message right there in the room.
The Magic Fold: Once the cleaning tool is built, it gets folded into the room's doorframe (the RMSNorm weights). This means when the real message travels through later, the cleaning happens automatically, with zero extra time or speed loss.

The Results: A Massive Improvement

The team tested this on three famous AI models: LLaMA-3-8B, Qwen-2.5-7B, and Mistral-7B.

The Old Way: When they tried to shrink LLaMA-3-8B to 3 bits using the standard method (RTN), the message became almost unintelligible, with a score (Perplexity) jumping from 5.83 to 14.09. That's a huge drop in quality.
The RDQ Way: Using their new "walk-through" method, they got the score down to 7.55. That is a 46.4% improvement over the old method!
At 4 bits: They also got the best results ever recorded, beating the previous champions by a wide margin (e.g., 5.62 vs 6.92 for LLaMA).

What This Means

The authors are very clear: this isn't just a better tool; it's a new way of seeing the problem. They proved that the "drift" is the main reason these models break at low bit-widths. They also showed that the old idea of "cleaning each room independently" is a dead end because it ignores the snowball of noise coming from the previous rooms.

While they can't fix the problem for 2-bit compression yet (the noise is just too strong there), they have shown that for 3-bit and 4-bit compression, we can keep the models smart and fast. The best part? This new method works with the standard tools engineers already use, so it can be deployed on phones and computers right now without needing any special, slow hardware.

In short: The tunnel was getting messy because we were cleaning the rooms in the wrong order. By cleaning them in the order the message actually travels, we can keep the signal clear all the way to the end.

Technical Summary: RDQ (Residual Distribution Quantization) for Large Language Models

1. Problem Statement: The Sub-4-bit Degradation Barrier

Post-training quantization (PTQ) of Large Language Models (LLMs) faces a sharp performance cliff when reducing precision below 4 bits. While 4-bit methods often approach FP16 quality, 3-bit quantization frequently results in catastrophic degradation. For instance, on LLaMA-3-8B, the Round-to-Nearest (RTN) method's perplexity (PPL) jumps from 5.83 at W4 to 14.09 at W3—a 2.42× increase for only a 1.33× reduction in bit-width.

The paper identifies the root cause of this failure as residual stream distributional drift. In transformer architectures, the residual stream $h_l$ is a shared representation read and written by every layer. When a layer $l$ is quantized, it introduces an error $\epsilon_l$ that persists into the residual stream $h_{l+1}$ . This error accumulates across layers, causing the distribution of activations in deeper layers to drift significantly from the original FP16 baseline.

Key empirical findings diagnosing this problem include:

Super-linear Drift Accumulation: The total Kullback-Leibler (KL) divergence between the quantized and FP16 residual distributions correlates with log-perplexity at a Pearson correlation of $r=0.999$ ( $p<0.001$ ) across all tested methods and bit-widths.
Non-Gaussian Distributions: 84% of layers in LLaMA-3-8B exhibit non-Gaussian residual distributions (Kolmogorov-Smirnov test, $p \le 0.05$ ), invalidating the Gaussian assumptions implicit in many existing PTQ theoretical analyses.
Variance Explosion: Per-layer residual stream variance grows by a factor of 6,548× across the depth of the model, indicating that later layers are categorically more sensitive to upstream quantization errors.

2. Methodology: Cascaded Error Compensation (CEC)

The proposed solution, RDQ (Residual Distribution Quantization), introduces a framework centered on Cascaded Error Compensation (CEC). Unlike prior PTQ methods (e.g., GPTQ, AWQ, SpinQuant) that optimize each layer independently against clean FP16 inputs, CEC acknowledges that during inference, a layer $l$ receives inputs that have already been distorted by the quantization of all upstream layers $0 \dots l-1$ .

The CEC Pipeline

CEC employs a sequential calibration procedure:

Sequential Processing: Layers are quantized one by one from $l=0$ to $L-1$ .
Drifted Activation Capture: Before quantizing layer $l$ , calibration data is passed through the model where layers $0 \dots l-1$ are already quantized. This captures the actual, drifted activation distributions ( $\hat{h}_l$ ) that layer $l$ will encounter at inference.
Per-Channel Scale Fitting: Using these captured drifted inputs, the method fits optimal per-channel scales using an AWQ-style grid search. The objective is to minimize the output reconstruction error given the specific drifted input distribution.
Mathematical Equivalence via Folding: The learned scales are folded into the preceding RMSNorm weights. This ensures the method is mathematically exact and incurs zero runtime overhead. The forward pass remains identical to a standard quantized model, compatible with existing inference stacks (Qualcomm AIMET, GGUF, etc.).

Key Distinction from Prior Work

The paper emphasizes that the critical innovation is not the scaling formula itself (which remains AWQ-style), but the calibration data source. Standard methods calibrate against idealized FP16 inputs, which do not exist at inference for deeper layers. CEC calibrates against the actual drifted distribution.

3. Key Contributions

Identification of Residual Stream Drift: The authors establish that residual stream drift is the dominant mechanism for sub-4-bit degradation, evidenced by the near-perfect correlation ( $r=0.999$ ) between KL drift and log-PPL.
Distributional Characterization: The paper documents that 84% of LLaMA-3-8B layers are non-Gaussian and that residual variance grows 6,548× across depth, challenging standard PTQ assumptions.
Cascaded Error Compensation (CEC): A sequential calibration procedure that corrects for cross-layer drift accumulation, achieving a 46.4% reduction in PPL compared to RTN at W3 on LLaMA-3-8B.
State-of-the-Art Performance: RDQ achieves the best results across three diverse architectures (LLaMA-3-8B, Qwen-2.5-7B, Mistral-7B) at both W3A16 and W4A16, outperforming the best published baselines (LeanQuant, SpinQuant) in every tested configuration.
Zero-Overhead Deployment: The output is standard group-128 asymmetric quantization with modified RMSNorm scalars, requiring no custom kernels or runtime overhead.

4. Experimental Results

The method was evaluated on WikiText-2 using 32 calibration sequences.

Performance Metrics (Perplexity):

LLaMA-3-8B:
- W3A16: 7.55 (vs. RTN 14.09, LeanQuant 7.61).
- W4A16: 5.62 (vs. FP16 5.557, LeanQuant 6.92).
Qwen-2.5-7B:
- W3A16: 7.46 (vs. RTN 8.20, LeanQuant 8.30).
- W4A16: 6.38 (vs. FP16 6.158, LeanQuant 7.25).
Mistral-7B:
- W3A16: 6.88 (vs. RTN 7.44, LeanQuant 7.30).
- W4A16: 5.73 (vs. FP16 5.632, LeanQuant 6.20).

Ablation Studies:

Sequential Calibration vs. Scaling: The paper demonstrates that sequential calibration alone (without AWQ scaling) actually degrades performance compared to RTN on Qwen (10.31 PPL vs. 8.20 PPL). Conversely, applying AWQ scaling to static FP16 calibration also underperforms (9.49 PPL). The combination of sequential drift-aware calibration and AWQ-style scaling is necessary for the observed gains.
Generalization: The method generalizes across architectures with different mechanisms (GQA, RoPE, Sliding Window), suggesting the drift-correction mechanism is architecture-agnostic.

5. Significance and Claims

The paper positions RDQ not merely as an engineering improvement in PTQ, but as a fundamental insight into the distributional dynamics of transformers under quantization.

Predictability: The authors claim that quantization quality is predictable from a single scalar (total KL drift), and that controlling this drift controls perplexity.
Failure Mode Correction: The work identifies that the dominant failure mode of sub-4-bit PTQ is not per-layer weight error, but the accumulation of cross-layer drift. Static calibration methods fail because they optimize for a distribution (clean FP16) that the layer never sees during inference.
Robustness: By fitting scales to the actual drifted activations, CEC sidesteps architecture-specific heuristics (like AWQ's salient channel detection) that may fail on models with different weight distributions (e.g., Qwen).
Theoretical Limitations: The paper modestly notes that while a theoretical bound exists for Gaussian regimes, it does not apply to the actual setting where 84% of layers are non-Gaussian. The observed performance gains are empirical, driven by the drift-correction mechanism rather than a tight theoretical bound.

In conclusion, RDQ establishes that addressing the "drift" in the residual stream is the key to unlocking robust 3-bit quantization, offering a scalable, zero-overhead solution that outperforms current state-of-the-art methods across multiple model families.