🤖 machine learning

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

SHRED is a novel, retain-set-free machine unlearning method for large language models that selectively removes memorized content by identifying and demoting high-information tokens via self-distillation, thereby achieving superior trade-offs between forgetting efficacy and model utility without requiring curated retain sets.

Original authors: Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei, Mohammad Rostami, Jesse Thomason, Robin Jia

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei, Mohammad Rostami, Jesse Thomason, Robin Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Unlearning" Dilemma

Imagine a large language model (LLM) as a super-photographic student who has read the entire internet. Sometimes, this student memorizes things they shouldn't have, like a celebrity's private address, a copyrighted book, or dangerous instructions on how to build a bomb.

To fix this, we usually need to "unlearn" that specific information. However, there's a catch:

The Old Way: To stop the student from remembering the bad info without making them forget everything else (like how to speak English or do math), we usually have to give them a "study guide" of safe, good examples to practice on while they forget the bad stuff.
The Problem: In the real world, we often don't have that perfect study guide. Creating one is hard, expensive, and can accidentally bias the student. Without it, trying to make the student forget usually breaks their brain, making them forget how to speak properly or refuse to answer harmless questions.

The Solution: SHRED

The authors propose a new method called SHRED (Self-distillation via High-surprisal-only Retain-free Entropy Demotion). That's a mouthful, so let's break it down with an analogy.

The Core Insight: Not All Words Are Created Equal

Imagine the student is reciting a story about a specific event they memorized:

"On July 4th, John Smith drove a red Ferrari to the Grand Canyon."

The paper discovered something fascinating about how the model "thinks" about this sentence:

The "Boring" Words: Words like "On," "the," "to," and "a" are very common. The model is very confident about these. They are like the scaffolding of a building.
The "Memorized" Words: Words like "July 4th," "John Smith," "red Ferrari," and "Grand Canyon" are specific facts. The model is actually less confident about these specific details compared to the boring words, because they are unique to that specific story.

SHRED's Big Idea: You don't need to scrub the whole sentence. You just need to target the specific, high-value facts (the scaffolding) and leave the common language structure alone.

How SHRED Works (The 2-Step Process)

SHRED is a "retain-set-free" method, meaning it doesn't need that external study guide. It uses the model's own brain as a teacher.

Step 1: The Detective Work (Selection)
The model reads the sentence it needs to forget. It looks at every word and asks: "How surprised am I by this word?"

If the model is not surprised (high probability), it's a common word (like "the"). SHRED ignores these.
If the model is surprised (low probability), it's a specific fact (like "John Smith"). SHRED marks these as the targets to forget.

Step 2: The Surgery (Demotion)
The model is then trained to do two things simultaneously:

Forget the Targets: It is forced to lower its confidence on the specific facts (e.g., "John Smith").
Keep the Scaffolding: It is told to keep its confidence high on the common words (e.g., "On," "the").

By doing this, the model learns to "forget" the specific secret without losing its ability to speak English. It's like removing the specific furniture from a room without tearing down the walls.

Why It's Better Than the Rest

The paper tested SHRED against many other methods on four different benchmarks (like a "final exam" for unlearning). Here is what they found:

The Pareto Frontier: Imagine a graph where the top-left corner is the "perfect score" (Total Forgetting + Total Utility). Most methods are stuck in the middle or bottom-right. SHRED is the only method that reaches the top-left corner without needing a study guide (retain set).
No "Brain Damage": Other methods often make the model start refusing to answer questions or speaking in gibberish. SHRED keeps the model's general intelligence intact.
Hallucination Fix: In a surprising twist, SHRED actually made the model less likely to make up fake facts about the real world. By removing the "fake" memorized stories, the model's natural knowledge shined through more clearly.
Robustness:
- Relearning: If you try to trick the model into remembering the secret again by showing it a few examples, SHRED's version of the model is much harder to trick than others.
- Privacy: It is very hard for hackers to tell if the model still "remembers" the secret data.
- Stability: You can train it for a long time without it breaking.

The "Knobs" and Settings

The authors found two main ways to tune SHRED:

The "How Much" Knob (P): You can choose to forget just the top 50% of the most specific words, or 100% of them. 50% seems to be the sweet spot for balancing forgetting with keeping the model smart.
The "Batch Size" Surprise: Usually, in AI training, using big groups of data at once is better. SHRED works best with tiny batches (even just one example at a time). It's like learning a new skill by practicing one specific move repeatedly rather than doing a whole workout; it helps the model "unlearn" more surgically.

Summary

SHRED is a clever way to make an AI forget specific secrets without needing a manual of "good examples" to guide it. It works by identifying the specific, unique words that hold the secret and gently lowering their importance, while leaving the common language structure untouched. The result is a model that has successfully forgotten the bad data but is still smart, helpful, and safe to use.

Technical Summary: SHRED – Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

1. Problem Statement

Large Language Models (LLMs) memorize training data, posing risks related to privacy leakage, copyright infringement, and the dissemination of hazardous knowledge. Machine unlearning aims to selectively remove this specific knowledge without the prohibitive cost of full retraining.

The standard unlearning paradigm requires a retain set (curated examples from the same domain as the forget set) to prevent catastrophic degradation of the model's general utility. However, in practice, constructing an ideal retain set is often infeasible due to data scarcity or the introduction of unwanted domain biases. Existing methods that attempt to bypass the retain set dependency often suffer from significant utility collapse, leading to compromised linguistic structures or excessive refusal rates. The core challenge is to achieve effective forgetting of specific memorized content while preserving general model capabilities without access to external retain data.

2. Methodology: SHRED

The authors propose SHRED (Self-distillation via High-surprisal-only Retain-free Entropy Demotion), a method that eliminates the need for a retain set by leveraging the model's own output distribution as a teacher. The approach is grounded in the insight that not all tokens within a "forget-set" instance carry memorized information equally.

Key Insight: Token-Level Information Density

In a model that has memorized a specific document, tokens representing high-information content (e.g., specific names, dates, unique relations) are assigned lower autoregressive probabilities compared to tokens representing general language scaffolding (e.g., punctuation, common phrases, stop words). The latter are supported by broad pretraining statistics, while the former are specific to the memorized instance.

The SHRED Procedure

SHRED operates in four stages using a single forget set ( $D_f$ ):

Teacher Forward Pass: The model to be unlearned (parameters $\theta$ ) acts as a frozen, off-policy teacher. It processes sequences from $D_f$ to generate per-token autoregressive probabilities $p_\theta(x_t | x_{<t})$ .
Selection of Forget Positions:
- The system excludes the initial context window (e.g., the question in QA or document prefix) to avoid damaging basic language modeling.
- It sorts the remaining tokens by probability in ascending order.
- The bottom- $P$ fraction (lowest probability, highest Shannon information) is selected as the forget set ( $F$ ). These positions contain the document-specific, high-information tokens.
- The remaining positions ( $R = T \setminus F$ ) serve as an implicit retain set, preserving the model's general language competence.
Construction of KL Targets:
- Forget Positions ( $t \in F$ ): The target distribution is modified to "demote" the memorized token.
  - Variant A (Retrieval Unlearning): Only the specific memorized token $x_t$ is demoted (logits set to $-\infty$ ).
  - Variant B (Knowledge Unlearning): The memorized token and the top- $\pi$ nucleus of the probability distribution are demoted to prevent paraphrasing.
- Retain Positions ( $t \in R$ ): The target distribution remains identical to the teacher's original distribution, acting as an anchor to preserve utility.
Top-K KL Self-Distillation: The student model ( $\theta'$ ) is trained to minimize the KL-divergence between its predictions and the constructed target distribution $q_t$ over a top- $K$ vocabulary subset. This single objective simultaneously drives the model away from high-information memorized content and anchors it to general language patterns.

3. Key Contributions

Retain-Set-Free Unlearning: SHRED is the first method to demonstrate that effective unlearning can be achieved without any external retain data, relying instead on the internal structure of the forget set itself.
Selective Logit Demotion: By identifying and demoting only high-surprisal (high-information) tokens while preserving low-surprisal tokens, SHRED avoids the utility collapse associated with uniform unlearning approaches.
Unified Objective: The method utilizes a single self-distillation objective that naturally balances forgetting and utility preservation, removing the need for complex hyperparameter balancing between forget and retain loss terms.
Robustness: The approach is shown to be robust against membership inference attacks (MIA), relearning attacks, and sequential continual unlearning scenarios.

4. Experimental Results

The authors evaluated SHRED across four standard benchmarks: TOFU (fictitious author QA), MUSE (News/Books), RWKU (Real-world entities), and Hubble (YAGO/Gutenberg).

Pareto Optimality: SHRED establishes a new Pareto frontier on all four benchmarks, outperforming retain-set-dependent baselines (such as GradDiff, NPO, and DPO) in the trade-off between forget efficacy and model utility.
Forget Efficacy: SHRED matches the performance of retrained "Target" models (oracles trained without the forget data) on forget metrics without exhibiting the "over-forgetting" seen in methods like Gradient Ascent.
Utility Preservation: Unlike baselines that often cause utility collapse or require a retain set to maintain performance, SHRED maintains or even improves general utility. Notably, on the TOFU benchmark, SHRED was observed to actively recover world knowledge that had been hallucinated or suppressed by the initial fine-tuning process, reducing hallucinations on real-world questions where the "Target" model still failed.
Robustness:
- MIA: Small-batch SHRED training achieves privacy leakage scores close to the retrained oracle, significantly outperforming baselines.
- Relearning: SHRED models are resistant to "relearning" attacks where the model is briefly fine-tuned on the forget set again; they recover knowledge much slower than baselines.
- Continual Unlearning: SHRED degrades utility slowly over multiple rounds of unlearning, whereas baselines like NPO and RMU suffer significant utility drops.
Hyperparameters: The method is most effective with a demotion percentage ( $P$ ) of 50–75% and small batch sizes (BS=1 or 2), which help the model escape local minima in the loss landscape.

5. Significance and Claims

The paper claims that SHRED represents a significant step forward in the practical deployment of machine unlearning. By removing the dependency on a curated retain set, SHRED addresses a major bottleneck in real-world applications where such data is unavailable or difficult to curate.

The authors emphasize that SHRED's ability to use the model's own "benign" tokens as anchors allows it to preserve general capabilities naturally, avoiding the "catastrophic forgetting" of general utility often seen in unlearning tasks. Furthermore, the observation that SHRED can reverse hallucinations introduced by fine-tuning suggests that the method does not merely suppress knowledge but can refine the model's distribution to be more aligned with general world knowledge.

The paper concludes that SHRED offers a Pareto-optimal trade-off between forgetting and utility, is robust against various attack vectors, and remains stable under overtraining and continual unlearning scenarios, making it a viable solution for privacy and safety compliance in LLMs without the overhead of data curation.