💬 NLP

Quantization-Robust LLM Unlearning via Low-Rank Adaptation

This paper proposes a quantization-robust unlearning method using Low-Rank Adaptation (LoRA) that freezes the base model to preserve unlearning updates under aggressive 4-bit post-training quantization, thereby significantly improving utility and reducing privacy leakage compared to standard full-parameter fine-tuning.

Original authors: João Vitor Boer Abitante, Joana Meneguzzo Pasquali, Luan Fonseca Garcia, Ewerton de Oliveira, Thomas da Silva Paula, Rodrigo C. Barros, Lucas S. Kupssinskü

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: João Vitor Boer Abitante, Joana Meneguzzo Pasquali, Luan Fonseca Garcia, Ewerton de Oliveira, Thomas da Silva Paula, Rodrigo C. Barros, Lucas S. Kupssinskü

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Eraser" That Gets Smudged

Imagine you have a giant, incredibly smart library (a Large Language Model or LLM) that has read millions of books. However, some of those books contain secrets, private information, or copyrighted stories that the library owner is legally required to remove.

Machine Unlearning is the process of trying to "erase" those specific books from the library's memory without destroying the rest of the library.

Usually, librarians (the algorithms) do this by making tiny, delicate adjustments to the library's catalog. They whisper to the system, "Forget this specific page," and the system makes a microscopic change to its internal notes.

The Twist:
To make the library run fast and cheap on regular computers, we often compress the catalog into a smaller, simpler format called Quantization. Think of this like translating a high-definition movie into a low-resolution, grainy video to save space.

The Disaster:
The paper discovered a major flaw: When you try to erase a secret using those tiny, delicate adjustments, and then compress the library into that grainy video, the erasure disappears.

It's like trying to draw a tiny, precise line on a piece of paper with a fine-tip pen, and then photocopying that paper on a broken, low-quality machine. The machine is so "grainy" that it can't see your tiny line. It just sees the original picture. The "forgetting" gets smudged out, and the secret comes back.

The Solution: The "Sticky Note" Strategy

The authors propose a new way to do the erasing called LoRA (Low-Rank Adaptation).

Instead of trying to rewrite the entire library catalog with tiny, invisible changes, they suggest a different approach:

Freeze the Library: Don't touch the original books or the main catalog at all. Keep them exactly as they are.
Add a "Sticky Note" System: Instead of changing the books, we attach a small, separate set of "sticky notes" (adapters) to the shelves.
Write Big: Because we aren't trying to be delicate with the main catalog, we can write on these sticky notes with bold, thick markers. We make the changes huge and obvious.
The Magic Merge: Before we compress the library (quantize it), we tape these sticky notes directly onto the books.

Why this works:
When the low-quality photocopier (the 4-bit quantization) tries to scan the library, it might miss a tiny pencil mark. But it cannot miss a thick, bold marker. The "forgetting" signal is now so strong and structural that even the grainy compression can't erase it.

The Results: A Win for Privacy and Speed

The researchers tested this on a famous AI model (Llama-2) using two types of data: news articles and Harry Potter books.

The Old Way (Full Fine-Tuning): When they tried to erase the data and then compress the model to 4-bit, the model remembered the secrets perfectly. The erasure failed.
The New Way (LoRA): When they used the "sticky note" method, the model successfully forgot the secrets, even after being compressed.
- Privacy: The model stopped leaking private info.
- Utility: The model didn't get "dumb." It still knew how to write good stories and answer questions about things it wasn't supposed to forget.

The Takeaway

If you want to build an AI that is safe, private, and runs on cheap, fast devices (like phones or small servers), you can't just use the old "delicate eraser" method. You need to use LoRA.

Think of it this way: If you want to remove a stain from a rug, scrubbing gently might not work if you're going to vacuum it immediately after. But if you use a heavy-duty stain remover that changes the color of the fabric significantly, the vacuum won't be able to "un-see" the change.

This paper proves that by making the "forgetting" changes bigger and more structural, we can ensure that AI models stay safe and private, even when we shrink them down to fit on small devices.

1. Problem Statement

Large Language Models (LLMs) require Machine Unlearning to remove specific sensitive, private, or copyrighted data from their training sets. However, practical deployment often necessitates Post-Training Quantization (PTQ) (e.g., reducing precision to 4-bit) to save memory and improve inference speed.

The core problem identified is that standard unlearning methods (like Gradient Ascent or Negative Preference Optimization) typically induce minimal weight updates across the full parameter space to preserve general utility. When these small updates are subjected to aggressive low-bit quantization (specifically 4-bit), the changes are often smaller than the quantization step size. Consequently, the quantization process "masks" the unlearning updates, causing the model to revert to its pre-unlearning state and recover the forgotten knowledge.

2. Methodology: Quantization-Robust Unlearning via LoRA

The authors propose a novel framework that integrates Low-Rank Adaptation (LoRA) into the unlearning process to overcome the limitations of full-parameter fine-tuning (Full-FT) under quantization.

Core Mechanism

Instead of updating all model weights, the method:

Freezes the pre-trained base model weights ( $W_0$ ).
Introduces trainable low-rank adapters ( $A$ and $B$ ) where the weight update is represented as $\Delta W = \frac{\alpha}{r} BA$ .
Optimizes only the adapter parameters to minimize the unlearning loss.
Merges the trained adapters into the base weights before applying quantization.

Why It Works (Theoretical Basis)

The paper identifies two key mechanisms that make LoRA robust to quantization:

Optimization Dynamics & Larger Step Sizes: Because LoRA restricts optimization to a low-rank subspace, it acts as an implicit regularizer. This allows the use of significantly higher learning rates (e.g., $10^{-4}$ vs. $10^{-7}$ in Full-FT) without catastrophic utility collapse. These larger steps generate weight updates ( $\Delta W$ ) that are large enough to cross quantization bin boundaries, ensuring the change survives the discretization process.
Magnitude Control via Architecture: The scaling factor ( $\alpha$ ) and the concentration of updates on specific layers (e.g., MLP or Attention projections) allow the method to amplify the unlearning signal. By focusing the "unlearning budget" on specific modules rather than diffusing it across the entire network, the magnitude of the update in those layers becomes sufficient to resist 4-bit quantization noise.

3. Key Contributions

Analysis of Failure Modes: The paper theoretically and empirically demonstrates that the conflict between the "minimal weight change constraint" of standard unlearning and the coarse resolution of 4-bit quantization leads to unlearning failure.
Proposed Framework: Introduction of a quantization-robust unlearning framework using LoRA, leveraging rank constraints and scaling factors to generate structural updates that survive aggressive quantization.
Empirical Validation: Comprehensive evaluation showing that LoRA-based unlearning significantly outperforms Full-FT in preserving unlearning effects after 4-bit PTQ, while maintaining model utility.

4. Experimental Results

The study was conducted on Llama-2-7B using the MUSE benchmark (datasets: BOOKS and NEWS). The methods evaluated included Gradient Ascent (GA) and Negative Preference Optimization (NPO), combined with utility regularization (GDR and KLR).

Key Findings:

Full-FT Failure: Standard Full-FT unlearning methods largely failed under 4-bit quantization. For example, on the BOOKS dataset, the utility preservation score for GA+GDR dropped from 68.74 (Full Precision) to 53.79 (4-bit), and forgetting signals were often lost.
LoRA Success:
- Utility Preservation: LoRA significantly reduced the utility drop caused by quantization. For GA+GDR on BOOKS, the 4-bit utility with LoRA was 53.16, compared to 53.79 for Full-FT (a smaller relative drop from the full-precision baseline). On NEWS, GA+GDR improved from 40.06 (Full-FT) to 44.82 (LoRA) under 4-bit quantization.
- Forgetting & Privacy: LoRA maintained strong forgetting capabilities. For GA+KLR on BOOKS, the Privacy Leakage (PrivLeak) improved from -25.68 (Full-FT) to -5.86 (closer to the ideal 0), indicating the unlearned model was indistinguishable from a model that never saw the data.
- Stability: In the most stable configurations (e.g., NPO+KLR on BOOKS), metrics remained nearly identical between full precision and 4-bit quantization when using LoRA, whereas Full-FT showed significant degradation.

5. Significance

This work addresses a critical gap in the deployment of privacy-compliant LLMs. As edge devices and resource-constrained environments increasingly rely on 4-bit quantization for efficiency, standard unlearning techniques become ineffective.

The paper demonstrates that parameter-efficient fine-tuning (LoRA) is not just a tool for adaptation but a viable strategy for robust unlearning. By concentrating the unlearning signal into a low-rank subspace, practitioners can deploy quantized models that genuinely forget sensitive data without reverting to their original state, enabling secure and efficient LLM deployment in real-world scenarios.