💻 computer science

What to Forget in Unlearning? Forget Set Curation for Language Models

This paper introduces CleanSlate, a benchmark demonstrating that the curation of forget sets is a critical upstream challenge in machine unlearning, where simple selection methods fail to suppress targeted content while overly aggressive strategies cause significant collateral damage to model capabilities.

Original authors: Animesh Jha, Arpandeep Khatua, Youssef Allouah, Sanmi Koyejo

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Animesh Jha, Arpandeep Khatua, Youssef Allouah, Sanmi Koyejo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the vast libraries of human knowledge are compressed into a single, incredibly smart computer program. This program, known as a language model, learns by reading trillions of words from books, websites, and songs. It becomes so good at this that it can mimic the style of a poet or finish a sentence from a famous novel. But sometimes, the owners of these programs need to remove specific information. Perhaps a song was used without permission, or a private story was accidentally memorized. The goal is to make the computer "unlearn" that specific piece of content without having to delete its entire memory and start over. This process is called machine unlearning. For years, researchers have focused on the mechanics of how to delete the data once they know exactly which words to target. They have treated the list of words to be forgotten as a simple starting point, assuming that if you tell the computer to forget a specific sentence, it will simply do so.

However, a new study from Stanford University suggests that this assumption misses a crucial, difficult step. The researchers realized that in the real world, a request to forget something rarely comes with a precise list of words. A user might ask a model to stop singing a specific song, but that song exists in the computer's memory in many different forms. It might appear in the official lyrics, but also in a news article quoting a line, a fan forum discussing the melody, a code file storing the text, or a review that mentions the chorus. The computer does not just know the song from one perfect copy; it knows it from a scattered cloud of these various mentions. The new research asks a fundamental question: how do you find all those scattered pieces of a song or book inside a massive library of text, so you can tell the computer to forget them? The team calls this missing step "forget set curation," and they found that getting it wrong can cause the computer to lose much more than just the unwanted song.

To investigate this, the researchers built a testing ground called CLEANSLATE. They gathered thousands of songs from the Billboard Hot 100 charts spanning from 1970 to 2025, along with fifty books. They then tested how well different methods could identify the specific text fragments that needed to be removed. They compared two main approaches. The first approach relied on standard search tools, similar to how a person might use a search engine to find a book. These tools looked for exact matches of words or phrases. The second approach was more aggressive: it looked directly at the specific parts of the text that the computer was most likely to reproduce and selected those for removal, essentially bypassing the search step to target the exact problem areas.

The results revealed a surprising complexity. When the researchers used the standard search tools to find the text, the results were often weak. The computer continued to sing the song or recite the book, even after the search tools had removed the obvious copies. This happened because the computer had learned the song from many small, scattered sources that the search tools missed. The "footprint" of a song is not just the official lyrics; it is a diffuse network of quotes, references, and fragments spread across the internet. When the researchers tried to be more thorough and targeted the exact windows of text that the computer used to generate the song, they achieved a different kind of problem. They successfully stopped the computer from singing the song, but in doing so, they damaged the computer's ability to remember other things. The model became worse at answering questions about the song's facts, and it also lost some of its general skills, such as its ability to reason or solve math problems.

The study showed that the choice of which text to remove is not just a technical detail; it is a major part of the solution itself. The researchers found that the same list of text to be forgotten could lead to very different outcomes depending on which algorithm was used to delete it. In some cases, a method that worked perfectly for one computer model caused significant damage to another. This means that you cannot simply pick a list of words and assume the deletion will be clean. The process of selecting the data and the process of deleting it are deeply connected. If you select too little, the unwanted behavior remains. If you select too much, or the wrong kind of text, you risk erasing valuable knowledge and harming the computer's general intelligence.

Ultimately, the research suggests that the idea of a simple, perfect list of things to forget is an illusion. In the messy reality of how these models learn, a song or a book is supported by a vast, invisible web of text. To truly unlearn a piece of content without hurting the rest of the model's mind, the selection of data must be done with extreme care, taking into account exactly how the specific computer model learned that content. The study concludes that practical unlearning cannot be solved by just improving the deletion tools; it requires a new way of thinking about how we find and choose the data in the first place. The path forward involves designing a system where the selection of what to forget and the method of forgetting are planned together, ensuring that the computer can let go of a specific request without losing its ability to understand the world around it.

Technical Summary: Forget Set Curation for Language Models

Problem Statement

Machine unlearning aims to remove targeted data or behaviors from a trained model without retraining from scratch. However, existing benchmarks and methods typically assume that the "forget set"—the specific examples used to drive the unlearning update—is already known and provided. In realistic deployment scenarios, a requester may ask a model to stop reproducing a specific song, book, or protected work without knowing which specific spans, documents, or near-duplicates within a trillion-token corpus support that behavior.

This paper identifies a missing upstream problem: forget set curation. This is the process of mapping a high-level suppression request (e.g., "stop reproducing Never Gonna Give You Up") to the concrete data spans that should be passed to an unlearning algorithm. The challenge is that the evidence supporting a model's verbatim continuation is rarely confined to a canonical copy of the work. Instead, it forms a "corpus footprint" distributed across lyric aggregators, fan forums, code snippets, news articles, and incidental discussions. A curator must recover enough of this footprint to suppress the target behavior without causing collateral damage to non-requested content or general model capabilities.

Methodology: CLEANSLATE Benchmark

To study this problem, the authors introduce CLEANSLATE, a benchmark and evaluation protocol designed to evaluate the full pipeline from suppression request to curated forget set to edited model.

Dataset and Scope

Content: CLEANSLATE covers 4,616 Billboard Hot 100 songs (1970–2025) and 50 books (mostly from Project Gutenberg).
Models: Evaluation spans six models ranging from 7B to 32B parameters (including LLaMA-3.1, OLMo, Gemma, Nemotron, and Qwen).
Extraction Profiles: The benchmark measures "extractability" using probabilistic extraction. A work is considered extractable if the model assigns a high probability ( $p \geq 0.001$ ) to a target suffix given a prefix. Profiles are generated per-model to account for varying memorization levels.
Content-Grounded QA: To distinguish between suppressing verbatim reproduction and erasing factual knowledge, the authors constructed 12,088 QA pairs. These test whether a model can still answer factual questions about a work (e.g., plot details, author, year) after unlearning, even if it can no longer recite the text.

Evaluation Pipeline

Curator ( $A$ ): Receives a suppression request ( $W_f$ ) and a corpus ( $C$ ), then outputs a forget set ( $D_f$ ) and optionally a retain set ( $D_r$ ).
Unlearner ( $U$ ): A fixed unlearning algorithm updates the model $\theta$ to $\theta'$ using $D_f$ and $D_r$ .
Evaluation: The resulting model is scored on:
- Target Suppression ( $F$ ): Reduction in extractable windows for the forget set.
- Collateral Suppression ( $R$ ): Unintended reduction in extractability for the retain set.
- Capability Retention: Performance on content-grounded QA and general benchmarks (GSM8K, BBH, WinoGrande, CoQA, HumanEval+, LAMBADA).

Curators Tested

The study compares three types of curators:

BM25-pre / BM25-mid: Lexical retrieval using BM25 indices over pretraining and midtraining corpora.
Infini-gram-mid: Exact-substring retrieval using an Infini-gram-mini index to find maximal character spans.
Evaluation-Aware (EA): A theoretical baseline that bypasses retrieval and directly selects the specific evaluation windows known to be extractable, serving as a robustness test for the evaluation protocol rather than a practical solution.

Key Results

1. Diffuse Corpus Footprints

The study confirms that cultural works have "diffuse corpus footprints." Older works spread outward through quotes and discussions, while newer works often inherit language through idioms and stock phrases (inward diffusion).

Outward Diffusion: A song like "Smile" (2024) achieved 100% coverage at the 5-gram level in the Common Crawl snapshot, with spans appearing in code, reviews, and forums, not just lyric sites.
Inward Diffusion: Newer songs often contain phrases that predate the song itself (e.g., quoting a 1910 poem), meaning the supporting evidence may not look like a copy of the target work at all.

2. Failure Modes of Retrieval-Based Curators

Natural lexical and exact-substring curators (BM25, Infini-gram) yield forget sets that lead to weak suppression.

Model Dependency: The effectiveness of retrieval-derived forget sets is heavily model-dependent. Larger models (e.g., Gemma-3-12B) showed some movement, while smaller models (e.g., OLMo-3-7B) barely moved.
Lack of Selectivity: Even when suppression occurred, it was rarely selective. For example, LLaMA-3.1-8B lost between 51% and 55% of its retain-pool extractability while gaining at most 29% suppression on the forget pool.

3. The Selectivity Gap in Evaluation-Aware Curation

When the curator bypasses retrieval and selects the exact windows used for evaluation (EA curator):

High Suppression: Target suppression reached near 100% across all models.
Severe Collateral Damage: This approach caused substantial collateral suppression (80–100% reduction in retain-pool extractability) and significant capability regressions.
Algorithm Interaction: The degree of capability loss varied by unlearning algorithm. For instance, using EA inputs with SimNPO caused an average capability drop of 5.1 percentage points, whereas UNDIAL largely preserved capability (+0.1 pp) despite similar suppression levels.

4. The Interdependence of Curation and Unlearning

The study demonstrates that the forget set and the unlearning algorithm interact strongly. The same forget set produced vastly different outcomes depending on the unlearner (e.g., SimNPO vs. RMU vs. UNDIAL).

No Universal Solution: Neither retrieval-based nor evaluation-aware curation achieved selective unlearning (high target suppression with low collateral damage) across the tested grid.
Pipeline Nature: Practical unlearning cannot be treated as a modular process where curation is a fixed premise. The choice of data determines both what is unlearned and what is damaged.

Significance and Claims

The paper argues that forget set curation should be treated as one of the objects of evaluation in machine unlearning, not merely a preprocessing step.

Reframing the Problem: The authors claim that practical unlearning is not just an optimization problem once a forget set is given; the data chosen for forgetting fundamentally dictates the trade-off between suppression and utility.
Limitations of Current Benchmarks: Existing benchmarks that assume a known forget set obscure the critical upstream challenge of mapping real-world requests to data.
Future Direction: The paper suggests that future curators must be "algorithm-aware," potentially combining corpus-footprint signals with model-specific extraction profiles and influence estimation to construct forget sets that localize effects.

The authors conclude that while CLEANSLATE studies a narrow setting (verbatim suppression of songs and books), the core insight holds broadly: the mapping from request to data is a critical, non-trivial component of the unlearning pipeline that requires joint evaluation with the unlearning algorithm.