🤖 machine learning

Unforgettable Generalization in Language Models

This paper investigates how language models behave when "unlearning" skills via fine-tuning on randomized labels, revealing that the generalization of forgetting is highly unpredictable and task-dependent, often failing to extend beyond specific training examples despite superficially random outputs, while underlying task capabilities remain intact as evidenced by successful linear probing.

Original authors: Eric Zhang, Leshem Choshen, Jacob Andreas

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eric Zhang, Leshem Choshen, Jacob Andreas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who has read almost everything on the internet. It can write poems, solve math problems, and tell you why the sky is blue. But sometimes, this robot learns things you don't want it to know, or it gets too good at a specific trick that you want to un-teach it. This is the world of "machine unlearning," a branch of artificial intelligence research asking a tricky question: Can we make a robot truly forget?

To understand the experiment, you need to know two things. First, these robots (called Large Language Models) are trained by reading tons of text and then "fine-tuned" on specific examples to get better at a job, like answering science questions. Second, a common way to try to make them forget is to show them the same questions again but with the answers completely scrambled or made up. The idea is that if you confuse the robot enough with wrong answers, it might just give up on that topic entirely. The big mystery researchers have been chasing is: Does this confusion spread? If you confuse the robot about one specific question, does it forget how to answer all similar questions, or does it just get confused about that one specific example?

The Great "Un-Training" Experiment

In this paper, researchers Eric Zhang, Leshem Choshen, and Jacob Andreas from MIT decided to play a game of "un-learning" with a giant language model called Llama2. They treated the model like a student who had just aced a test, and then they tried to make it forget the material by re-teaching it with random, nonsense answers. They wanted to see if the student would forget the whole subject or just the specific practice problems they were shown.

The Big Surprise: It Depends on the Subject
The researchers tested this on 21 different types of tasks, ranging from "physical commonsense" (like figuring out which object is heavier) to "entailment" (deciding if one sentence logically follows another). The results were wild and unpredictable.

Sometimes, the forgetting worked perfectly. If they tried to make the model forget how to do entailment classification (a logic puzzle about sentences), the model would go blank. It would start giving random, useless answers to new questions it had never seen before. It was as if the robot had lost the ability to do that specific type of logic entirely.

But other times, the forgetting was a total failure. When they tried to make the model forget physical commonsense (like how gravity works) or science questions, the model was stubborn. Even after being trained on random, wrong answers, it would still answer new science questions correctly! It was like a student who was forced to memorize the wrong answers for a specific practice test, but when the real test came around, they still knew the right answers. The model seemed to have forgotten the specific examples it was trained on, but it kept its general knowledge intact.

It's Not About How Hard the Test Is
You might think the model only forgets easy things and remembers hard things. The researchers checked this, and it turns out, that's not true. They found that the difficulty of the task didn't predict whether the model would forget. For example, the model was actually better at remembering the answers to the "hard" ARC Challenge questions than the "easy" ARC Easy questions, even though both were science questions. The difficulty of the question didn't matter; the type of question did.

The Secret Clues: Confidence and "Brain" Variability
So, what makes a skill easy or hard to forget? The researchers found two clues hidden inside the model's "brain" (its internal data representations):

Confidence: If the model was already very unsure about the answer before they started the forgetting experiment, it was easier to make it forget. If the model was confident, it held on tighter.
Brain Variability: This is a bit like looking at how much the model's thoughts wiggle. If the model's internal "thoughts" (representations) for a task were all over the place (high variability), it was harder to erase. If the thoughts were very consistent and similar (low variability), the model was easier to "un-train."

The "Shallow" Forget
Here is the most mind-bending part. Even when the model seemed to have successfully forgotten a skill—giving random answers to everyone—the researchers found the information wasn't actually gone. They built a simple tool called a "linear probe" (think of it as a tiny, specialized detector) and looked at the model's internal brain waves.

Surprisingly, this detector could still read the correct answers perfectly, even though the model itself was acting like it knew nothing. It's like a student who pretends to be confused and gives random answers in class, but if you look at their notes, the correct answers are still written there in perfect handwriting. The "forgetting" was just a surface-level act; the deep knowledge was still there, waiting to be found.

The Bottom Line
The paper concludes that trying to make AI forget specific skills by just re-training it on random answers is a gamble. It works for some tasks (like logic puzzles) but fails for others (like science and common sense). Furthermore, even when it seems to work, the information isn't truly deleted; it's just hidden behind a layer of confusion. The researchers suggest that we need to understand much more about how these models learn and store information before we can reliably make them forget what we don't want them to know.

Technical Summary: Unforgettable Generalization in Language Models

Problem Statement

The paper addresses a critical gap in the field of machine unlearning for Large Language Models (LMs). While recent research has focused on developing techniques to remove specific skills or declarative knowledge from LMs—often via fine-tuning on randomized labels to "unlearn" a task—there is limited understanding of how this forgetting generalizes. Specifically, when an LM is trained to forget a capability on a specific training set, does its behavior change predictably on unseen examples of the same task, or does the forgetting remain localized only to the training data? The authors investigate whether targeted skill removal via fine-tuning results in robust, generalizable unlearning or if the model retains the ability to perform the "forgotten" task on new instances.

Methodology

The authors conduct a systematic study using the Llama2 7-billion parameter base model, with additional validation on GPT-J-6B and GPT-2. The experimental setup involves 21 diverse multiple-choice tasks spanning commonsense reasoning, reading comprehension, mathematics, toxicity detection, and entailment classification.

The core procedure follows a three-stage pipeline:

Pre-training: Starting with a pre-trained LM.
Fine-tuning: The model is fine-tuned on a target task ( $T_{train}$ ) to maximize accuracy, establishing a baseline of learned capability.
Forgetting: The model is further fine-tuned on the same training set ( $T_{train}$ ) but with labels chosen uniformly at random. This is a standard method for inducing forgetting.

Metrics:
To quantify the success of forgetting, the authors define two metrics:

Forget Gap: The difference between the accuracy after forgetting and the expected random accuracy (50% for binary tasks). A gap of 0 implies complete forgetting.
Forget Ratio: A normalized measure comparing the drop in accuracy after forgetting against the total possible gain from fine-tuning. A ratio of 1 indicates complete forgetting; 0 indicates no forgetting.

Analysis Dimensions:
The study evaluates forgetting across several dimensions:

Task-Level Generalization: Comparing forgetting performance across different task types.
Cross-Task Forgetting: Fine-tuning on random labels from Task A and evaluating on Task B.
Predictors of Forgetting: Analyzing correlations between forgetting success and task difficulty, model confidence, and the variance of hidden state representations.
Learning vs. Forgetting Order: Investigating the relationship between the order in which examples are learned and the order in which they are forgotten.
Depth of Forgetting: Training linear probes on the model's hidden representations after the forgetting procedure to determine if the information is truly erased or merely censored.

Key Results

1. High Variability in Generalization

The primary finding is that the generalization of forgetting is highly task-dependent and unpredictable.

Robust Tasks: Tasks involving commonsense reasoning (e.g., PIQA, ARC) and scientific question answering (e.g., SciQ, PubMedQA) are remarkably resilient to forgetting. Even after training on random labels, these models often maintain high accuracy on test examples, effectively failing to generalize the "forgetting" behavior.
Fragile Tasks: Lower-level linguistic tasks, particularly entailment classification (e.g., MNLI, RTE, CoLA) and grammatical acceptability, are easily forgotten. Models trained on random labels for these tasks tend to produce near-random predictions on new test instances.

2. Evaluation Task Determines Success

In cross-task experiments, the authors found that the effectiveness of the forgetting procedure is largely determined by the task being evaluated, not the task used for training. For instance, training a model to forget commonsense reasoning often fails to induce forgetting on that specific task but may inadvertently cause the model to forget other, unrelated linguistic tasks.

3. Predictors of Forgetting

Task Difficulty: Contrary to intuition, task difficulty (measured by fine-tuned accuracy) is not a reliable predictor of forgettability. Harder tasks are not necessarily harder to forget.
Model Confidence: The paper explicitly states in its conclusion that dataset difficulty and model confidence are not predictive of whether a task is forgotten, despite observing a weak correlation in specific scatter plots.
Representation Variance: The total variance of the hidden states (specifically the last token of the question in the penultimate layer) is a strong predictor. Tasks with lower variance in their training data representations are associated with greater generalization of forgetting. This suggests that "broader" or more diverse tasks are harder to forget.

4. Shallow Nature of Forgetting

Even in cases where the model appears to have successfully forgotten a task (producing random outputs), the information is not truly removed. Linear probes trained on the model's hidden representations can still reliably recover the correct task labels with high accuracy. This indicates that fine-tuning on random labels induces shallow forgetting, effectively censoring the output without erasing the underlying capability from the model's representations.

5. Robustness Across Models and Methods

The observed trends hold across different model architectures (Llama2, GPT-J, GPT-2) and different forgetting strategies (random labels vs. flipped labels), suggesting these are fundamental properties of how fine-tuning interacts with task structures rather than artifacts of a specific model or method.

Significance and Claims

The paper claims to highlight the difficulty and unpredictability of performing targeted skill removal from language models via standard fine-tuning techniques.

Unpredictability: The authors emphasize that one cannot assume that training a model to forget a skill on a specific dataset will result in that skill being forgotten on new, similar instances. The behavior is governed by complex, structured cross-task variabilities.
Shallow Removal: The study challenges the efficacy of random-label fine-tuning as a true "unlearning" mechanism. Since linear probes can recover the forgotten capabilities, the method does not remove information from the model's internal representations, raising concerns about its utility for privacy or safety applications where true data removal is required.
Task Dependency: The findings suggest that the success of unlearning is intrinsic to the nature of the task being evaluated, rather than the specific training data used for the unlearning process.

The authors conclude that while fine-tuning on random labels is a simple and widely used method for forgetting, it results in highly variable and often superficial outcomes, underscoring the need for further research into the relationship between training data for forgetting and the generalization of model predictions.