🤖 machine learning

LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty

The paper introduces LoTUS, a novel and efficient machine unlearning method that eliminates training sample influence from pre-trained models by smoothing prediction probabilities to an information-theoretic bound, thereby outperforming state-of-the-art baselines on both standard and large-scale datasets without requiring retraining.

Original authors: Christoforos N. Spartalis, Theodoros Semertzidis, Petros Daras, Efstratios Gavves

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christoforos N. Spartalis, Theodoros Semertzidis, Petros Daras, Efstratios Gavves

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart student who has memorized a massive textbook. This student is so good that they can recite specific facts about every single page. But now, imagine a situation where a few pages of that textbook were written by someone who wants to remain anonymous, or perhaps those pages contain sensitive information that needs to be erased.

You want the student to "forget" those specific pages completely, as if they never saw them, without having to send the student back to school to re-read the entire book from scratch. That is the problem of Machine Unlearning.

The paper introduces a new method called LoTUS (Logits Tempering Unlearning Strategy) to solve this. Here is how it works, using simple analogies:

The Problem: The "Over-Confident" Student

Deep learning models (like the student) often "memorize" specific details about the data they are trained on. If you show them a picture of a cat with a unique scar, they don't just learn "cat"; they learn "cat with that specific scar."

Because of this memorization, the model becomes over-confident. It says, "I am 99.9% sure this is that specific cat!" This is dangerous because it reveals that the model has seen that specific image before. Hackers can use this over-confidence to figure out if a specific person's photo was in the training data (a privacy attack).

The Solution: LoTUS (The "Cool-Down" Strategy)

Instead of retraining the whole model (which is like making the student re-read the whole library), LoTUS gently "cools down" the model's confidence on the specific images it needs to forget.

Think of the model's output as a hot, sharp laser beam. LoTUS turns that laser into a soft, diffuse spotlight.

Smoothing the Probabilities: The method takes the model's predictions for the "forget" images and makes them less certain. It spreads the confidence out so the model isn't so sure about those specific details anymore.
The "Gold Standard" Goal: The goal is to make the model act on those "forget" images exactly how it would have acted if it had never seen them. If the model had never seen the "scarred cat," it would be less sure about it. LoTUS forces the model to be less sure, mimicking that "never seen" state.
The Temperature Knob: The method uses a "temperature" setting (like a thermostat).
- If the model is still too confident (too hot), the system turns up the temperature to make the predictions more random and uncertain.
- If the model gets too confused (too cold), it turns the temperature down to sharpen the focus again.
- It constantly adjusts this knob until the model's behavior on the "forget" images matches the behavior of a model that never saw them.

The New Tool: The "No-Retest" Scorecard

Usually, to prove you successfully made a student forget something, you have to send them back to school, re-teach them everything without the bad pages, and compare the two versions. This is expensive and slow.

The authors invented a new way to check the work called RF-JSD (Retrain-Free Jensen-Shannon Divergence).

The Analogy: Instead of re-teaching the student and comparing the two versions, you just look at the student's current answers and compare them to a "mystery box" of new, unseen questions.
If the student's answers to the "forget" questions look similar to their answers to the "new" questions (where they have no memory), you know they have successfully forgotten.
This allows you to check for privacy leaks without needing to retrain the model, which is crucial for huge datasets like ImageNet where retraining is impossible.

What They Found

The researchers tested LoTUS on various models (like Vision Transformers and ResNet) and datasets (from small ones like CIFAR to the massive ImageNet).

Better than the rest: LoTUS was faster and more effective than all the other methods they compared it against.
It works on big data: They successfully used it on ImageNet (over 1 million images), a scenario where retraining is practically impossible.
It avoids the "Streisand Effect": Sometimes, when you try to hide something, you make it stand out more. LoTUS avoids this; it doesn't just make the model guess wrong (which would be obvious); it makes the model genuinely uncertain, which is the natural state of not knowing.

In Summary

LoTUS is a clever "eraser" for AI. It doesn't delete the whole book; it just gently blurs the specific pages that need to be forgotten, ensuring the AI doesn't get too confident about them, all while saving massive amounts of time and computer power. They also built a new ruler (RF-JSD) to measure how well the erasing worked without needing to rebuild the whole machine.

Technical Summary: LoTUS – Large-Scale Machine Unlearning with a Taste of Uncertainty

1. Problem Statement

Machine Unlearning (MU) aims to remove the influence of specific training samples (the "forget set") from a pre-trained model without retraining the model from scratch. This is critical for privacy protection, copyright compliance, and bias mitigation. However, existing methods face three primary challenges:

Efficiency: Retraining from scratch (the "gold standard") is computationally prohibitive, especially for large-scale datasets like ImageNet1k.
Data Accessibility: In real-world scenarios, full access to the original training data is often restricted due to privacy policies or storage limitations, making retraining impossible.
Effectiveness vs. Retention: Methods must effectively eliminate the influence of forget samples while preserving the model's utility on the remaining data. Existing approaches often struggle to balance these competing objectives or fail to approximate the gold standard's performance accurately.

Current unlearning techniques generally fall into exact unlearning (statistically indistinguishable from retraining but often infeasible) or approximate unlearning (using various metrics to gauge proximity to the gold standard). A significant gap exists in methods that can operate efficiently on large-scale models under realistic data constraints while providing robust evaluation metrics that do not require a retrained gold standard model.

2. Methodology: LoTUS

The authors propose LoTUS (Logits Tempering Unlearning Strategy), a novel approach that operates directly in the model's output space using an information-theoretic framework.

Core Concept: Entropy and Uncertainty

LoTUS addresses the tendency of Deep Neural Networks (DNNs) to memorize sample-specific features, leading to over-confident predictions on training data. This over-confidence is a vulnerability exploited by Membership Inference Attacks (MIAs). LoTUS seeks to smooth the prediction probabilities of forget samples to increase their entropy, making the model's response to these samples resemble its response to unseen data.

Theoretical Framework

The method is grounded in the separation of global information (shared features across a class, e.g., general cat shape) and subset-specific information (unique features of specific samples, e.g., unique fur patterns).

Objective: Eliminate subset-specific information ( $I(f_{orig}(X_f); X_f | X)$ ) while retaining global information ( $I(f_{orig}(X_f); X)$ ).
Approximation: Since estimating mutual information directly is computationally intensive, LoTUS approximates the unlearning objective by equating the accuracy of the unlearned model on the forget set ( $D_f$ ) to the accuracy of the original pre-trained model on an unseen set ( $D_u$ ):
$Acc(f_{un}, D_f) = Acc(f_{orig}, D_u)$
Here, $D_u$ can be a validation set or synthetic data. This ensures the unlearned model does not retain the "memorized" confidence it had on $D_f$ .

Implementation: Dynamic Temperature and Gumbel Noise

LoTUS employs a knowledge distillation framework where a student model ( $f_{un}$ ) learns from a teacher model ( $f_{orig}$ ).

Gumbel-Softmax Perturbation: The teacher's logits are perturbed using Gumbel noise ( $g \sim \text{Gumbel}(0,1)$ ) and a temperature parameter ( $\tau$ ) via the Gumbel-Softmax function. This introduces stochasticity and allows for differentiable sampling.
Dynamic Temperature Adjustment: The temperature $\tau_d$ $τ_{d}$ is dynamically adjusted in each epoch based on the accuracy gap between the student on the forget set and the teacher on the unseen set:
$\tau_d = \exp(\alpha \cdot (Acc(f_{un}, D_f) - Acc(f_{orig}, D_u)))$
- If the student is too accurate on $D_f$ (indicating memorization), $\tau_d > 1$ , smoothing the probabilities to increase entropy.
- If the student is too inaccurate (over-unlearning), $\tau_d < 1$ , sharpening probabilities to restore utility.
Loss Function: The loss function combines a "forget" term (using the perturbed teacher output with dynamic $\tau_d$ ) and a "retain" term (using the teacher output with $\tau \to 0$ to sharpen predictions and preserve utility on the retain set).

3. Key Contributions

LoTUS Algorithm: The first unlearning method operating directly in the output space using an information-theoretic bound. It controls uncertainty via a dynamic temperature parameter, avoiding indiscriminate entropy maximization.
RF-JSD Metric: The introduction of the Retrain-Free Jensen-Shannon Divergence (RF-JSD). This metric evaluates unlearning effectiveness by comparing the unlearned model's output on the forget set against the original model's output on an unseen set. It eliminates the need for a retrained gold standard model, making it suitable for real-world, large-scale scenarios. RF-JSD shows a strong Pearson correlation ( $0.92 \pm 0.04$ ) with the established JSD score.
Large-Scale Experimental Setup: A novel evaluation framework utilizing the ImageNet1k dataset (1.2M samples) with restricted data access, simulating conditions where retraining is infeasible.

4. Experimental Results

The authors evaluated LoTUS on Vision Transformer (ViT) and ResNet18 architectures across five datasets: CIFAR-10, CIFAR-100, MUFAC, TinyImageNet, and ImageNet1k.

Effectiveness: LoTUS consistently outperformed eight state-of-the-art baselines (including Finetuning, NegGrad+, SalUn, and SCRUB) in terms of the Avg Gap metric (balancing forgetting and retention) and JSD (measuring distributional similarity to the gold standard and resilience to the Streisand effect).
Efficiency: LoTUS demonstrated superior computational efficiency, often requiring less time than baselines. This is attributed to its ability to use only a small percentage of retain samples (30%) and its linear complexity in auxiliary computations.
Large-Scale Performance: On ImageNet1k, where retraining was impossible, LoTUS achieved the lowest RF-JSD score (1.11 vs. 1.22 for the original model, and higher for baselines), indicating effective unlearning. It also uniquely preserved Retain Accuracy while significantly reducing Membership Inference Attack (MIA) accuracy.
Robustness: The method remained effective even when the unseen set consisted of synthetic AI-generated data (CIFAKE), validating the relaxation of the distributional similarity assumption.

5. Significance and Claims

The paper claims that LoTUS represents a significant advancement in machine unlearning by:

Bridging the Gap: Providing a method that is both effective and efficient for large-scale models where retraining is impractical.
Real-World Applicability: Introducing the RF-JSD metric enables rigorous evaluation in scenarios where the gold standard model is unavailable, a common constraint in privacy-sensitive industries.
Theoretical Rigor: Moving beyond heuristic approaches by grounding the unlearning process in information theory, specifically targeting the removal of subset-specific information while preserving global utility.
Superiority: Demonstrating that LoTUS surpasses existing state-of-the-art methods in balancing the trade-off between forgetting specific data and maintaining model performance, while also mitigating the Streisand effect (where unlearning inadvertently makes data more identifiable).

The authors note that while the framework extends to class unlearning, their primary experimental validation focuses on instance-wise unlearning. The work is presented as a step toward scalable, privacy-preserving machine learning in constrained environments.