🤖 machine learning

When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

This paper demonstrates that 8-bit quantization, by acting as implicit regularization to prevent overfitting, significantly outperforms full-precision models in continual learning scenarios, enabling large language models to retain knowledge more effectively with minimal replay buffers.

Original authors: Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student (a Large Language Model) who is incredibly smart but has a terrible memory. Every time they learn a new subject, they tend to forget everything they learned in the previous subjects. This is called "catastrophic forgetting."

To help them remember, teachers usually use a "replay" method: they give the student a few flashcards from old lessons while teaching new ones. The bigger the stack of flashcards, the better the student remembers.

However, in the real world, we often need to shrink these students down to make them faster and cheaper to run on regular computers. This process is called quantization. Think of it like translating a high-definition, 4K movie into a grainy, black-and-white 8-bit video game. You lose some detail, but the file size becomes tiny.

The common belief was: "If you shrink the student (quantize them), they will forget even faster, so you need more flashcards to help them remember."

This paper flips that idea on its head. Here is what the researchers found, using simple analogies:

1. The "Noisy" Student vs. The "Perfect" Student

The researchers tested three types of students:

The Perfect Student (FP16): Has a crystal-clear, high-precision brain.
The 8-bit Student (INT8): A slightly compressed version.
The 4-bit Student (INT4): A very compressed, "grainy" version.

The Surprise: When these students learned a new subject (like Math) without any help, the "Perfect Student" actually forgot their old skills (like reading comprehension) the fastest. The "grainy" students (especially the 4-bit ones) held onto their old memories much better.

Why? The researchers suggest that the "grainy" students have a little bit of static noise in their brains. This noise acts like a gentle shaker. When the student tries to learn something new, the noise prevents them from over-focusing too hard on the new lesson and erasing the old one. It's like trying to draw a new picture on a canvas that's slightly bumpy; you can't press too hard, or the paint will smear. This "bumpiness" (noise) actually helps them keep the old picture visible.

The "Perfect Student," having no noise, is too smooth. They can learn the new lesson perfectly, but in doing so, they completely wipe the old picture off the canvas.

2. The Flashcard Experiment (Replay Buffers)

The team then gave these students different amounts of "flashcards" (replay buffers) from their old lessons while teaching them new ones. They tested how many flashcards were needed to keep the old skills alive.

The Result: Even a tiny stack of flashcards (just 0.1% of the old data) made a huge difference for everyone.
The Winner: The 8-bit Student (INT8) turned out to be the "Goldilocks" of the group. They didn't need a huge stack of flashcards, but they didn't forget as easily as the Perfect Student either. They found the perfect balance between learning new things and remembering old things.
The 4-bit Student: They were great at remembering, but they needed a slightly larger stack of flashcards to perform well on the new tasks.

3. The "Less is More" Rule

The paper's main takeaway is that being less precise can actually make a model smarter at learning continuously.

For Reading (NLU): You don't need many flashcards at all. Even the compressed models work great with a tiny stack.
For Math and Coding: You need a moderate stack of flashcards (5–10%).
The Big Reversal: If you use the "Perfect Student" (high precision), they forget old skills very quickly unless you give them a massive stack of flashcards. If you use the "8-bit Student," they remember old skills much better, even with a smaller stack of flashcards.

Summary

Think of it like packing for a trip.

High Precision (FP16): You pack everything perfectly organized. But if you add one new item, you have to rearrange the whole suitcase, and you often lose the old items.
Quantized (INT8/INT4): You pack with a little bit of "mess" (noise). This mess actually acts as a buffer. When you add a new item, the mess prevents the old items from being pushed out completely.

The Bottom Line:
If you want an AI that can learn new things over time without forgetting the old stuff, you don't always need the most powerful, high-precision computer. Sometimes, a slightly "compressed" (8-bit) model with just a few flashcards from the past works better than a super-precise model with a massive memory bank. The "noise" in the compressed model is actually a feature, not a bug.

Technical Summary: When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

Problem Statement

Continual learning in Large Language Models (LLMs) faces the fundamental challenge of catastrophic forgetting, where models lose previously acquired capabilities when fine-tuned on new data. While replay-based methods (reintroducing samples from prior tasks) are a standard mitigation strategy, they introduce a trade-off between retention and computational/storage costs.

Concurrently, the practical deployment of LLMs necessitates quantization (compressing model parameters to lower precision, e.g., INT8 or INT4) to improve efficiency. However, the interaction between quantization precision and replay buffer strategies remains underexplored. Conventional wisdom suggests that higher precision (FP16) is always preferable for stability, yet it is unclear how quantization noise affects the dynamics of forgetting and retention in sequential learning scenarios.

Methodology

The authors conducted a systematic investigation using the LLaMA-3.1-8B model across three precision levels: FP16, INT8, and INT4.

Adaptation Strategy: All experiments utilized Low-Rank Adaptation (LoRA) with rank $r=8$ $r = 8$ and $\alpha=16$ $α = 16$ . Weights were frozen, and only adapters were learned.
- FP16/INT8: Standard inference-time weight quantization with high-precision LoRA adapters.
- INT4: QLoRA-style NF4 quantization.
Continual Learning Protocol: The study followed the LoRI benchmark protocol, training for exactly one epoch per dataset in three sequential stages:
1. Stage A: Train on 100% Natural Language Understanding (NLU) data (aggregation of 8 datasets: BoolQ, PIQA, SocialIQA, ARC-Challenge, ARC-Easy, OpenBookQA, HellaSwag, Winogrande).
2. Stage B: Train on 100% Mathematical Reasoning (GSM8K) while interleaving replay buffers of varying sizes ( $B \in \{0, 0.1, 0.5, 1, 2, 5, 10, 20\}\%$ ) from NLU data.
3. Stage C: Train on 100% Code Generation (CodeAlpaca) while interleaving replay buffers from both NLU and GSM8K data.
Evaluation: Performance was measured via Forward Accuracy (performance on the current task immediately after training) and Backward Accuracy (retention of performance on previous tasks).

Key Contributions and Results

1. The "Less is More" Phenomenon

Contrary to the assumption that higher precision yields better continual learning, the study reveals a striking inversion:

Initial Performance: FP16 achieved superior initial task performance (74.44% on NLU).
Continual Retention: Quantized models significantly outperformed FP16 on final task forward accuracy for subsequent tasks.
- NLU Retention: Quantized models showed 8–15% higher retention on final tasks compared to FP16.
- Code Generation: INT4 models achieved nearly double the performance of FP16 (40% vs. 20%).
The Role of Replay: While minimal replay (0.1%) dramatically improved retention across all precision levels (e.g., increasing NLU retention after Math training from 45% to 65%), INT8 consistently offered the optimal balance between learning plasticity and knowledge retention.

2. Quantization as Implicit Regularization

The authors hypothesize that the superior performance of quantized models stems from quantization-induced noise acting as implicit regularization.

FP16 Dynamics: The high-precision model is more susceptible to overfitting on new task gradients. Its stability allows new updates to overwrite prior knowledge more cleanly, exacerbating catastrophic forgetting.
Quantized Dynamics: The noise introduced by INT8/INT4 quantization may smooth the loss landscape, biasing the model toward flatter minima that generalize better. This effect potentially amplifies the influence of the few replayed samples, helping to anchor the model to previously learned knowledge.

3. Practical Guidelines for Deployment

The study provides empirical guidelines for balancing memory and accuracy in compressed continual learning:

NLU Tasks: Small replay buffers (1–2%) suffice across all precision levels.
Math & Code Tasks: Moderate buffers are beneficial.
- Math: 5–10% replay is recommended; for INT8/INT4 models prioritizing math, this should rise to 10–20%.
- Code: INT4 models benefit from 5–10% replay; enlarging to 10–20% improves stability for INT8 and FP16 settings.
General Finding: Quantized models require less replay than FP16 to achieve comparable retention, challenging the notion that higher precision is always necessary for stability.

Significance and Limitations

Significance:
This work challenges the conventional wisdom that higher precision is inherently superior for continual learning. It demonstrates that INT8 quantization offers a dual benefit: computational efficiency and superior continual learning dynamics. The findings suggest that the "noise" of quantization can be a feature rather than a bug, acting as a regularizer that prevents the overfitting to new gradients that plagues high-precision models.

Limitations:
The authors explicitly note several constraints:

Scope: Evaluation is restricted to LLaMA-3.1-8B and a specific set of tasks (NLU, Math, Code), which may not generalize to other architectures or domains like multi-modal learning.
Quantization Levels: Only uniform quantization levels (FP16, INT8, INT4) were tested; mixed-precision or adaptive strategies were not explored.
Statistical Rigor: Experiments lacked multi-seed runs and confidence intervals.
Replay Strategy: Simple reservoir sampling was used; more sophisticated selection strategies (e.g., herding, clustering) were not investigated.

The paper concludes by providing a benchmark framework for future studies and practical guidelines for deploying compressed models in real-world continual learning scenarios.