💻 computer science

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

This paper argues that conventional accuracy and perplexity metrics fail to capture the behavioral divergence caused by post-training quantization in large language models, introducing "correctness agreement" to reveal that even moderate quantization induces significant structural distortions—particularly in query and key projections—that create an illusion of equivalence between base and quantized models.

Original authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, hyper-smart robot chef (a Large Language Model) that can write stories, solve riddles, and cook up ideas. Now, imagine you want to shrink this robot down so it fits in your pocket. To do that, you try to simplify its recipe book by rounding off all the tiny, precise measurements to whole numbers. This process is called quantization.

For a long time, people checking if the robot still works just asked: "Does it still get the right answer?" and "Does it sound smooth?" They used standard tests like accuracy (how often it's right) and perplexity (a fancy score for how confused the robot sounds). If the robot got the same score, everyone assumed it was the exact same robot, just smaller.

But this new paper says: Hold on! That's an illusion.

The "Look-Alike" Trap

The authors, Baha Rababah, Cuneyt Gurcan Akcora, and Carson K. Leung, discovered that a robot can get the same test score as the original but actually be making completely different decisions to get there.

Think of it like two students taking a math test.

Student A (The Original): Solves the problem step-by-step and gets the right answer.
Student B (The Quantized Version): Guesses the right answer by luck or uses a totally different, weird method.

If you only look at the final grade (the score), they look identical. But if you look at how they solved it, they are totally different. The paper introduces a new way to check called Correctness Agreement. This metric asks: "Did both students get the same specific questions right?"

The results? Even when the "grades" (accuracy) looked fine, the "agreement" dropped. The simplified robot was often getting the right answer for the wrong reasons, or missing the specific questions the original robot nailed.

The "Tipping Point" in the Recipe

The researchers tested this on four different robot chefs (Llama-3.2-3B, Vicuna-7B, Mistral-7B, and Llama-3.1-8B) and tried shrinking them from 8-bit down to 2-bit.

They found a "tipping point" where things start to break:

8-bit to 5-bit: The robot's internal recipe book stays mostly the same. The flavors (statistics) are preserved.
4-bit: This is where the trouble starts. The recipe begins to warp.
3-bit and 2-bit: This is the "breakdown zone." The robot's internal structure gets distorted. The "flavors" of the numbers get weird, with some becoming too heavy or too light.

The "Sensitive Parts" of the Brain

Here is the most interesting part: The robot's brain isn't made of identical parts. The paper found that some parts are way more fragile than others.

Imagine the robot's brain has four main teams working together:

The Question Team (Query/Q)
The Key Team (Key/K)
The Value Team (Value/V)
The Output Team (O)

When the researchers squeezed the robot down to low bits (like Q3 K or Q2 K), the Question and Key teams got totally scrambled. Their internal numbers went wild, changing shape and size drastically. But the Value and Output teams stayed surprisingly calm and stable, even when the others were panicking.

This means that if you want to shrink a robot without breaking it, you can't just squeeze everything equally. You have to be careful with the Question and Key teams, because they are the most sensitive to being squished.

The "Smooth Voice" Lie

One of the biggest surprises was about perplexity. This is a score that measures how "smooth" or "confused" the robot sounds. The paper found that a robot could have a very smooth voice (low perplexity) even when its internal brain was completely distorted and it was making different decisions than the original.

It's like a person who speaks perfectly clearly but is saying things that don't make sense to the original speaker. The paper shows that perplexity is not a reliable sign that the robot is still thinking like the original. You can have a smooth voice and a broken brain at the same time.

What They Actually Found (and What They Didn't)

The authors didn't just guess; they measured this across multiple models and datasets like WikiText-2, HellaSwag, and ARC. They showed that:

Aggressive shrinking (3-bit and 2-bit) causes a "non-linear" break. It's not a slow slide; it's a sudden cliff where the robot's behavior changes drastically.
Correctness Agreement is a better way to check if a robot is truly the same, because it catches those "lucky guesses" that accuracy misses.
Q4 K seems to be the "safe zone" where the robot is still mostly okay, but Q3 K is where the degradation really starts to show up in the robot's decision-making.

So, the next time you see a tiny, super-efficient robot model, don't just trust the scorecard. The paper suggests that even if it looks perfect on paper, it might be a different robot entirely, making different choices and thinking in a completely different way than the giant version it came from. The illusion of equivalence is just that—an illusion.

Technical Summary: The Illusion of Equivalency in LLM Quantization

Problem Statement

Post-training quantization (PTQ) is a standard technique for deploying Large Language Models (LLMs) in resource-constrained environments by reducing the numerical precision of parameters and activations. However, current evaluation practices rely almost exclusively on aggregate metrics such as accuracy and perplexity. The authors argue that these metrics fail to capture behavioral changes induced by quantization. Specifically, models with similar accuracy or perplexity scores may exhibit substantial divergence in their predictions on individual examples. This creates an "illusion of equivalency" where a quantized model appears functionally identical to its base counterpart, despite underlying structural distortions and decision-level inconsistencies.

Methodology

The study formalizes post-training quantization as an operator family acting on model parameters and analyzes its joint impact on internal attention weights and decision behavior. The research employs a two-stage analytical approach:

Structural Analysis: The authors treat quantization as a structural operator on attention projection matrices ( $W^Q, W^K, W^V, W^O$ ). They quantify structural deviations using:
- Statistical Functional: Moments (mean, standard deviation, skewness, kurtosis) computed on weight distributions.
- Distributional Divergence: Metrics including Cosine similarity, Euclidean distance, Kolmogorov–Smirnov (KS) statistic, and Kullback–Leibler (KL) divergence between base and dequantized weight vectors.
- Granularity: Analysis is performed at the model level (aggregated across all blocks) and the layer level (isolating Q, K, V, and O projections).
Behavioral Evaluation: The study introduces Correctness Agreement (CA), a decision-level metric defined as the fraction of examples where both the base model and the quantized variant produce correct predictions. Unlike aggregate accuracy, CA measures the overlap in correct decisions, disregarding cases where both models fail.

Experimental Setup:

Models: Llama-3.2-3B, Vicuna-7B-v1.5, Mistral-7B-v0.1, and Llama-3.1-8B.
Quantization Schemes: Implemented via llama.cpp, covering Legacy methods (Q8_0, Q5_0, Q4_0) and K-quantization methods (Q6_K through Q2_K).
Datasets: WikiText-2 and C4 for perplexity; HellaSwag, Winogrande, and ARC for downstream zero-shot task performance.

Key Contributions

Formalization of Quantization as an Operator: The work models PTQ as a transformation $T_c(\theta)$ and studies its joint effect on attention projection weights and decision behavior.
Statistical and Divergence-Based Sensitivity Analysis: The authors propose a framework to quantify layer-wise structural drift using statistical moments and distributional distances, revealing non-linear breakpoints at low bit-widths.
Introduction of Correctness Agreement (CA): A new metric that complements accuracy and perplexity by measuring the consistency of correct predictions between the base and quantized models.
Comprehensive Empirical Sweep: To the authors' knowledge, this is the first controlled study sweeping from 8-bit to 2-bit across both legacy and K-quantization schemes, linking block-level attention projection drift to decision behavior changes.

Experimental Results

1. Statistical and Structural Distortions

High/Moderate Bit-Widths (8-bit to 5-bit): Quantization largely preserves statistical properties (near-zero skewness, stable kurtosis, baseline standard deviation). Legacy quantization (e.g., Q5_0) shows minimal structural impact.
Low Bit-Widths (4-bit to 2-bit): Aggressive quantization, particularly Q3_K and Q2_K, introduces pronounced distortions. These include increased skewness, reduced kurtosis, systematic mean shifts, and higher variance.
Layer Sensitivity: The Query (Q) and Key (K) projections are consistently more sensitive to quantization than Value (V) and Output (O) projections. Under aggressive compression (Q3_K, Q2_K), Q and K layers exhibit volatile skewness and collapsing kurtosis, while V and O remain relatively stable until the lowest bit-widths.
Divergence Metrics: Cosine similarity degrades gradually, while KL divergence and KS statistics show sharp increases at lower bit-widths, indicating significant redistribution of weight mass.

2. Behavioral Evaluation

Perplexity and Accuracy: These surface-level metrics often remain stable under moderate quantization (e.g., Q4_K, Q5_K) or show non-monotonic behavior. In some cases, lower bit-widths (e.g., Q3_K) yield lower perplexity without guaranteeing behavioral consistency.
Correctness Agreement (CA): CA is consistently lower than aggregate accuracy and decreases as precision drops. Even when accuracy appears preserved, the overlap in correct predictions between the base and quantized models drops sharply. This indicates that quantization alters which examples the model gets right, even if the total count of correct answers remains similar.

Significance and Claims

The paper claims to expose a fundamental disconnect between conventional performance metrics and the actual behavioral fidelity of quantized models.

The Illusion of Equivalency: The authors demonstrate that high accuracy or low perplexity does not guarantee that a quantized model preserves the base model's decision logic. A model can appear performant while fundamentally diverging in its specific predictions.
Non-Linear Breakpoints: The study identifies specific thresholds where structural integrity collapses. Q4_K is identified as the upper bound of "safe" quantization, Q3_K marks the onset of significant degradation, and Q2_K represents a breakdown regime where structural and behavioral consistency is severely compromised.
Component Vulnerability: The findings highlight that Q and K projections are the primary bottlenecks for low-bit quantization, suggesting that future quantization strategies should consider adaptive precision allocation (e.g., preserving higher precision for Q/K while compressing V/O).
Evaluation Paradigm Shift: The work motivates a shift in evaluation beyond aggregate metrics, advocating for behavioral evaluation (like Correctness Agreement) and structural analysis to ensure that quantized models are not merely statistically similar but behaviorally aligned with their base counterparts.