Remaining-data-free Machine Unlearning by Suppressing Sample Contribution

This paper proposes MU-Mis, a remaining-data-free machine unlearning method that achieves performance comparable to data-dependent approaches by theoretically linking sample contribution to model sensitivity and directly suppressing this sensitivity using only the pre-trained model.

The Gist

Imagine you have a brilliant student who has studied hard for a final exam. They know the answers to every question because they memorized the entire textbook. Now, imagine the student has a legal right to "be forgotten." They want to forget a specific chapter about "Quantum Physics" so that if someone asks them about it later, they genuinely don't know it.

However, there's a catch: The student cannot be allowed to forget anything else. They must still ace the questions about "History," "Biology," and "Math."

This is the challenge of Machine Unlearning.

The Old Way: The "Scorched Earth" Strategy

Most existing methods try to make the student forget the chapter by confusing them.

• The Analogy: Imagine a teacher telling the student, "From now on, whenever you see a picture of a cat, call it a 'dog'." Or, "Let's just pretend cats don't exist."
• The Problem: This is messy. By messing with the student's brain to make them forget cats, you accidentally mess up their ability to recognize dogs, tigers, or even other animals. The student becomes confused and performs poorly on the rest of the exam.
• The Fix: To fix this, the teacher has to spend hours re-teaching the student using all the other books (the remaining data) to restore their knowledge. This is slow, expensive, and sometimes, the teacher doesn't even have access to the other books anymore.

The New Way: MU-Mis (The "Silent Nudge")

The paper you shared introduces a new method called MU-Mis. Instead of confusing the student, it uses a very clever, precise trick based on sensitivity.

The Core Insight: "How much does the model care?"

The researchers discovered something fascinating: When a model (or student) learns a specific piece of data, it becomes hyper-sensitive to it.

• The Analogy: Think of the student's brain as a house of cards. When they learn a specific fact (like a specific cat photo), they build a tiny, very sensitive tower of cards just for that fact. If you touch that specific card, the whole tower wobbles.
• The Discovery: The researchers found that you can tell if the student has "learned" something just by seeing how much their brain shakes when they look at that specific thing. If the model is "sensitive" to the data, it means the data contributed to the learning.

The Solution: "Turn Down the Volume"

Instead of confusing the student or re-teaching them, MU-Mis simply turns down the volume on that specific sensitivity.

• The Metaphor: Imagine the student is listening to a radio. The "Quantum Physics" chapter is playing at a very loud, annoying volume (high sensitivity). The "History" chapter is playing at a normal volume.
• The Action: MU-Mis doesn't smash the radio or change the station. It just finds the specific volume knob for "Quantum Physics" and turns it all the way down until it's silent.
• The Result: The student now genuinely doesn't know Quantum Physics (the volume is zero), but because they didn't smash the radio, the "History" and "Math" channels are still loud and clear. The student's performance on the rest of the exam remains perfect.

Why This is a Big Deal

1. No "Cheat Sheet" Needed: Most methods need to look at the other data (the History and Math books) to make sure the student doesn't forget them. MU-Mis is so precise that it doesn't need to see the other books at all. It just tweaks the specific "Quantum" sensitivity.
2. Speed: Because it doesn't need to re-teach the student, it's incredibly fast. It's like turning a knob versus rewriting a whole textbook.
3. Safety: It doesn't leave "ghosts" of the forgotten data. The student truly forgets the chapter, protecting privacy, without ruining their other skills.

In a Nutshell

Previous methods tried to make the AI forget by scrambling its brain and then hoping to fix it later. This new method, MU-Mis, acts like a scalpel. It identifies exactly where the AI "remembers" the data (by measuring how sensitive it is) and gently suppresses that specific memory, leaving everything else perfectly intact.

It's the difference between trying to erase a word from a page by scribbling over the whole page (old way) versus using a magic eraser that only removes that one word without touching the rest of the sentence (MU-Mis).

Technical Summary

Here is a detailed technical summary of the paper "Remaining-Data-Free Machine Unlearning by Suppressing Sample Contribution" (MU-Mis), published as a conference paper at ICLR 2026.

1. Problem Statement

Machine Unlearning (MU) aims to remove the influence of specific training samples (forgetting data, $D_f$) from a pre-trained model to comply with the "right to be forgotten," while preserving the model's utility on the remaining data ($D_r$).

• The Core Challenge: Existing MU methods often rely on heuristic strategies (e.g., random relabeling, knowledge distillation from "useless" teachers) because quantifying a specific sample's contribution to the learning process is difficult. These heuristics often cause catastrophic unlearning (severe degradation of utility on $D_r$) or over-forgetting.
• The Data Accessibility Bottleneck: Most high-performing MU methods require access to the remaining data ($D_r$) to fine-tune and recover utility. However, in many real-world scenarios (e.g., distributed data silos, privacy regulations, or massive datasets), accessing $D_r$ is infeasible or too costly.
• The Gap: There is a lack of Remaining-Data-Free (RDF) methods that can match the performance of state-of-the-art (SoTA) methods that do have access to $D_r$.

2. Methodology: MU-Mis

The authors propose MU-Mis (Machine Unlearning by Minimizing Input Sensitivity), a principled approach that operates solely on the pre-trained model without needing $D_r$.

A. Theoretical Foundation: Contribution via Input Sensitivity

The paper establishes a theoretical link between a sample's contribution to training and the model's input sensitivity.

• Hypothesis: A training sample $x_i$ contributes to the learning process by increasing the learned model's sensitivity to that specific input.
• Mathematical Derivation:
  • Let the training algorithm be $A$, mapping dataset $D$ to function $f$. The contribution of a sample $x_i$ is related to $\partial A / \partial x_i$.
  • Through Taylor expansion of Gradient Descent updates, the authors show that the change in the learned function $f$ due to $x_i$ is approximately proportional to the input sensitivity of the model at $x_i$, denoted as $\nabla_x f(x_i)$.
  • Specifically, the contribution manifests as a disproportionately higher sensitivity for the target class logit ($f_c$) compared to irrelevant class logits ($f_{c'}$).

B. Empirical Observation

Experiments confirm that:

1. Training Effect: After training, the input sensitivity ($\|\nabla_x f_c\|_F$) of the target class for training data increases significantly compared to random initialization.
2. Unlearning Signal: In a retrained model (where $D_f$ is removed), the sensitivity gap between the target logit and irrelevant logits for the forgetting data shrinks. The target sensitivity drops, and the irrelevant sensitivity rises (recovering to pre-training levels).

C. The Optimization Objective

Based on these findings, MU-Mis defines a loss function to suppress the sample contribution by minimizing the sensitivity disparity:
$$\mathcal{L}(D_f; w) = \frac{1}{N_f} \sum_{x_f \in D_f} \left( \|\nabla_x f_c(x_f, w)\|_F^2 - \|\nabla_x f_{c'}(x_f, w)\|_F^2 \right)$$

• Goal: Reduce the magnitude of the target logit's sensitivity ($\|\nabla_x f_c\|$) while simultaneously recovering the sensitivity of irrelevant logits ($\|\nabla_x f_{c'}\|$).
• Mechanism: This effectively "withdraws" the sample's contribution by reversing the specific sensitivity pattern induced during training, without introducing noise or requiring $D_r$.

D. Stopping Criterion

To prevent over-unlearning (which would harm utility), the algorithm uses a dynamic stopping rule:

• It monitors the ratio of the current irrelevant-class sensitivity to its initial value (in the pre-trained model).
• Optimization stops when $\|\nabla_x f_{c'}\|_F$ recovers sufficiently close to its initial level (controlled by a threshold $\delta$). This ensures the model's behavior on the remaining data is restored.

3. Key Contributions

1. Theoretical Insight: The first work to theoretically and empirically demonstrate that a sample's contribution to training is reflected in the amplified sensitivity gap between target and irrelevant logits. This allows contribution identification using only the pre-trained model.
2. Novel Algorithm (MU-Mis): Proposes a lightweight, principled unlearning method that directly suppresses sample contribution via input sensitivity minimization.
3. Performance Breakthrough: To the best of the authors' knowledge, MU-Mis is the first Remaining-Data-Free method to perform on par with top-performing Remaining-Data-Dependent methods.
4. Efficiency: The method is computationally efficient, avoiding the need for retraining or storing historical gradients/checkpoints.

4. Experimental Results

The authors evaluated MU-Mis on 6 datasets (CIFAR-10/100, TinyImageNet, PinsFaceRecognition, SVHN) across 3 tasks (Full-class, Sub-class, Random-subset) using ResNet-18 and ViT.

• Utility Preservation:
  • In Full-class and Sub-class unlearning, MU-Mis achieved the smallest Average Gap (difference from the retrained model) among all methods, outperforming SoTA RDF baselines (like JiT, SCAR, NG) and competing closely with or surpassing SoTA data-dependent methods (like SalUn, SCRUB, SSD).
  • It successfully preserved accuracy on remaining data (RA) and test data (TA) without ever seeing them.
• Privacy Protection:
  • MU-Mis achieved low Membership Inference Attack (MIA) scores, indicating successful removal of private information.
  • It showed the lowest KL divergence from the retrained model's output distribution on the forgetting data compared to other RDF methods.
• Efficiency:
  • MU-Mis was significantly faster than data-dependent methods. For example, on TinyImageNet with ViT, it reduced unlearning time from >1 hour (SalUn) to 3 minutes.
• Sequential Unlearning Resilience:
  • In sequential unlearning scenarios (multiple requests), MU-Mis demonstrated superior stability. Unlike other methods that suffered from "utility breakdown" or "knowledge residue" (recovery of forgotten classes), MU-Mis maintained consistent performance and indistinguishability from the retrained model.
• Random Subset Challenge:
  • While Random Subset unlearning remains the most difficult scenario (where forgetting and remaining data are highly entangled), MU-Mis still outperformed all existing RDF methods significantly, though it lagged slightly behind the specialized data-dependent method RUM.

5. Significance

• Practical Deployment: MU-Mis solves a critical bottleneck in real-world AI systems where retraining or accessing retained data is impossible due to cost, storage, or legal constraints.
• Paradigm Shift: It moves the field away from heuristic "confusion" strategies (random labeling) toward a principled, gradient-based mechanism grounded in the physics of learning (input sensitivity).
• Scalability: The method's efficiency makes it viable for large-scale models (like ViT) and continuous unlearning pipelines, offering a viable path for privacy-compliant AI in resource-constrained environments.

In summary, MU-Mis provides a robust, efficient, and theoretically grounded solution for machine unlearning that eliminates the dependency on remaining data while achieving performance comparable to methods that possess it.