💬 NLP

Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall

This paper investigates the impact of quantization on factual knowledge recall in large language models, revealing that while quantization generally causes information loss and performance degradation—particularly in smaller models—it does not always impair recall and can occasionally enhance it, with BitSandBytes demonstrating the highest preservation of original capabilities.

Original authors: Qianli Wang, Mingyang Wang, Nils Feldhus, Simon Ostermann, Yuan Cao, Hinrich Schütze, Sebastian Möller, Vera Schmitt

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qianli Wang, Mingyang Wang, Nils Feldhus, Simon Ostermann, Yuan Cao, Hinrich Schütze, Sebastian Möller, Vera Schmitt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly detailed library of facts inside a computer brain (a Large Language Model). This brain knows everything from "Who is Beyoncé's father?" to "What is the capital of France?"

Now, imagine you want to shrink this library down so it fits on a smaller shelf, making it faster and cheaper to run. This process is called quantization. It's like taking a high-resolution photo and compressing it into a smaller file size. Usually, you lose a little bit of detail in the process, but the picture still looks good.

This paper asks a very specific question: When we compress these AI brains, do they start forgetting the facts they used to know?

Here is what the researchers found, explained through simple analogies:

1. The "Shrinking" Process

Think of the AI model as a team of workers. In the original "full-precision" version, every worker has a high-end calculator and a thick notebook.
Quantization is like forcing the workers to use smaller, cheaper calculators and thinner notebooks.

The Goal: To save space and speed up the work.
The Risk: The workers might drop some information because their new tools aren't as precise.

2. The Main Discovery: "The Small Kids Forget More"

The researchers tested three different ways of compressing the models (like using different brands of cheap calculators) on three different AI models (two small ones and one medium-sized one).

The Finding: When you shrink the model, it generally loses some facts. However, the smaller models (like the 7B or 8B versions) are much more fragile.
The Analogy: Imagine a small child trying to carry a heavy backpack. If you make the backpack slightly heavier (or in this case, squeeze the information tighter), the child drops things. A larger adult (the bigger 14B model) can handle the squeeze much better and keeps their facts safe.

3. The Surprise: Sometimes Compression Helps!

You might think that making a tool "dumber" (less precise) always makes it worse. But the researchers found something weird: Sometimes, compressing the model actually made it remember facts better.

The Analogy: It's like a student who is so stressed by having too much information that they can't focus. If you give them a slightly simpler textbook (quantization), they might actually perform better because the "noise" is gone. The compression acted like a filter, helping the model focus on the right facts.

4. Where Does the Memory Go?

The researchers didn't just check the final answer; they looked inside the model to see where the memory was lost.

The "Last Layer" Problem: They found that the information loss mostly happens in the very last steps of the model's thinking process.
The Analogy: Imagine a relay race. The runners (layers) pass the baton (information) down the line. The researchers found that the baton usually gets dropped right at the finish line when the model is compressed. The early runners are fine, but the final runner struggles to hold onto the fact.

5. The "Best" Compression Method

The paper tested three different compression techniques:

GPTQ
AWQ
BitSandBytes (bib)

The Winner: BitSandBytes was the best at keeping the facts intact. It was like using a high-quality vacuum sealer that removed the air but kept the food fresh.
The Loser: Some methods, especially when compressing to very low precision (4-bit), caused the model to forget a lot, particularly for smaller models.

6. The "Bridge" Test

To test how well the models could connect dots (like: "Who is the mother of the singer of 'Superstition'?"), they used a "multi-hop" test.

The Result: Compression hurt the first step of the reasoning the most. If the model couldn't remember the first fact (who sings 'Superstition'), it couldn't solve the whole puzzle. However, if it got past the first step, it was surprisingly good at finishing the chain.

The Bottom Line

Compressing AI models is a bit like packing a suitcase for a trip.

If you pack too tight (high compression), you might leave some socks behind (forget facts).
Smaller suitcases (smaller models) are harder to pack without losing things.
Some packing methods (like BitSandBytes) are much better than others.
Occasionally, packing tighter helps you find what you need faster, even if you lost a sock or two.

Overall, the paper concludes that while compression does cause some information loss, it is still a very effective strategy. The models don't break; they just get a little "fuzzier" around the edges, and for many uses, that fuzziness is a fair trade for the speed and space savings.

1. Problem Statement

While quantization is widely adopted to accelerate Large Language Model (LLM) inference and reduce deployment costs, its impact on Factual Knowledge Recall (FKR) remains underexplored. Previous studies have examined quantization effects on multilinguality, bias, and alignment, but there is a lack of comprehensive understanding regarding how reducing parameter precision affects a model's ability to access and retrieve stored factual knowledge. Specifically, it is unclear whether quantization causes systematic information loss, how this loss manifests across different model architectures and layers, and whether lower-bit models consistently underperform their full-precision counterparts in knowledge-intensive tasks.

2. Methodology

The authors conducted a comprehensive empirical study combining performance benchmarking with interpretability-driven analysis.

Models & Quantization Techniques:
- Models: Llama3-8B, Qwen2.5-7B, and Qwen2.5-14B.
- Techniques: Three Post-Training Quantization (PTQ) methods were evaluated at various bit widths (4-bit and 8-bit):
  - GPTQ: Second-order Hessian-based optimization.
  - AWQ: Activation-aware weight quantization handling outliers.
  - BitSandBytes (bib): Integer quantization using optimized low-bit kernels (bib4 and bib8).
- Baseline: Full-precision models.
Datasets:
- LRE (Knowledge Probing): A dataset of knowledge triplets $(s, r, o)$ used to measure direct factual recall accuracy.
- TwoHop-Fact: A dataset designed to test Latent Multi-hop Reasoning (LMHR), requiring the model to bridge two facts (e.g., $e_1 \to e_2 \to e_3$ ) to answer a query.
Interpretability Analysis:
- Knowledge Memorization Analysis: Utilizing Knowledge Neuron Theory, the authors traced information loss to specific neurons and layers. They calculated "contribution scores" (increase in log-probability of correct tokens) for neurons in full-precision vs. quantized models to identify which neurons lose influence.
- Latent Multi-hop Reasoning Analysis: Evaluated using three metrics:
  1. Entity Recall Score (ENTREC): Ability to recall the bridge entity ( $e_2$ ) in a two-hop chain.
  2. Consistency Score (CNSTSCORE): Similarity between output distributions of one-hop and two-hop prompts.
  3. FKR Accuracy: Correct prediction of the final target object.

3. Key Contributions

Systematic FKR Evaluation: Provided the first comprehensive analysis of how different PTQ techniques and bit widths specifically degrade or preserve factual knowledge recall across multiple model families.
Layer- and Neuron-Level Insights: Mapped the degradation of knowledge to specific architectural components, revealing that quantization primarily impacts the final layers of the network where factual knowledge is often consolidated.
Counter-Intuitive Findings: Demonstrated that lower-bit quantization does not always lead to worse performance; in some cases, quantization acts as a regularizer, inadvertently enhancing factual recall.
Method Comparison: Identified BitSandBytes (bib) as the most effective technique for preserving FKR compared to GPTQ and AWQ, particularly in 8-bit configurations.

4. Key Results

A. General Performance Trends

Information Loss: Quantization typically causes information loss, leading to reduced FKR accuracy. This effect is amplified in smaller models (e.g., 7B/8B) within the same family compared to larger variants (14B).
Bit Width Sensitivity:
- 4-bit models: Consistently showed significant accuracy drops, especially with GPTQ and AWQ.
- 8-bit models: Generally retained performance comparable to full-precision models.
- BitSandBytes (bib): Showed the highest preservation of original FKR capabilities. In many cases, bib8 and even bib4 maintained accuracy close to full precision.

B. Granular Analysis (Neurons & Layers)

Layer-wise Impact: The degradation is not uniform. The study found that quantization primarily affects the last layers of the model.
- In Qwen2.5-7B, the most significant drop in neuron contribution scores occurred in the final two layers.
- In Llama3-8B, the decline occurred in middle-to-late layers, with some recovery in the final layers, suggesting architectural differences in how knowledge is stored.
Neuron Contribution: The number of "high-contributing" neurons (those critical for correct answers) decreased significantly after quantization, particularly for relations where the full-precision model had not yet saturated (e.g., "person father," "landmark on continent").

C. Multi-hop Reasoning (TwoHop-Fact)

First-Hop Sensitivity: Quantization disproportionately affects the first hop ( $r_1$ ) of reasoning chains. The degradation in the first hop strongly correlates (Spearman $\rho = 0.93$ ) with the failure to predict the bridge entity, which subsequently causes the second hop to fail.
Unpredictability: The impact of quantization is heterogeneous. While some quantized models performed worse, others (specifically bib variants) occasionally outperformed full-precision models, likely due to a regularization effect or beneficial noise injection.

D. Relation-Specific Fragility

Knowledge types that were not fully saturated in the full-precision model (e.g., specific biographical facts) were more fragile under quantization than common knowledge (e.g., "country capital city").

5. Significance and Implications

Deployment Viability: The study confirms that quantization remains a viable compression strategy for LLMs. While some accuracy loss is inevitable, it is generally modest and does not compromise the model's utility for factual tasks, especially when using 8-bit quantization or the BitSandBytes method.
Model Selection: For applications requiring high-fidelity factual recall, BitSandBytes (bib) is recommended over GPTQ or AWQ, particularly for 4-bit constraints.
Architectural Understanding: The findings suggest that factual knowledge is stored differently across model families (Llama vs. Qwen) and is concentrated in the later layers of the network. This provides a new target for future research into model compression and knowledge editing.
Paradoxical Improvement: The observation that quantization can sometimes improve recall challenges the assumption that compression is strictly detrimental, suggesting that controlled noise or regularization from quantization might help filter out spurious correlations.

Limitations

Language Scope: Experiments were limited to English datasets; multilingual FKR may be affected differently.
Model Size: Gradient-based analyses were restricted to models $\le$ 14B due to GPU memory constraints.
Quantization Scope: The study focused on weight-only PTQ. Activation quantization, KV cache compression, and Quantization-Aware Training (QAT) were not included.