🤖 AI

SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation

This paper introduces SUPREME, an open-source, multi-GPU framework designed to accelerate and standardize the reproducible evaluation of image unlearning methods by distributing computationally expensive training and testing stages across multiple accelerators.

Original authors: Petros Andreou, Jamie Lanyon, Axel Finke, Georgina Cosma

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Petros Andreou, Jamie Lanyon, Axel Finke, Georgina Cosma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a student who has studied a massive library of books to become an expert. Suddenly, the library owner says, "Please forget everything you learned about these specific five books." You can't just erase the pages from the student's brain; you have to teach them to un-learn that information without making them forget everything else they know.

This is the problem of Machine Unlearning. It's about removing the influence of specific data from an AI model without having to throw the whole model away and start from scratch.

The paper you provided introduces a new tool called SUPREME to help researchers figure out which "unlearning" techniques actually work best. Here is a simple breakdown of what they did and why it matters.

The Problem: The "One-Seed" Trap

Imagine you are testing a new recipe for a cake. If you bake it once and it tastes good, you might think the recipe is perfect. But what if that one time, you just got lucky with the temperature or the brand of flour? To be sure, you need to bake that cake ten times with different batches of ingredients to see if the recipe is consistently good.

In AI research, this "batch of ingredients" is called a seed (a random starting point).

The Issue: Most previous tools for testing unlearning could only run on one computer chip (one GPU) at a time. Because training AI is slow and expensive, researchers could only afford to run the test once or twice.
The Risk: If you only test once, you might get a lucky result that doesn't reflect reality. The paper argues that you need to run the test many times (across many "seeds") to know if a method truly works.

The Solution: SUPREME

The authors built SUPREME, which stands for Standardised Unlearning Platform for Reproducible Method Evaluation. Think of it as a high-speed, multi-lane race track for AI experiments.

Here is how it works using everyday analogies:

1. The Multi-GPU Highway
Previous tools were like a single-lane dirt road where only one car (one experiment) could go at a time. SUPREME is a multi-lane highway. It spreads the work across multiple graphics cards (GPUs) at once. This means researchers can run the same experiment ten times in the time it used to take to run it once. This allows them to see if the results are consistent or just a fluke.

2. The "Plug-and-Play" Toolbox
The framework is designed like a Lego set.

Researchers can snap in different datasets (like the "Pins Face Recognition" dataset used in the paper).
They can snap in different AI models (like ResNet18 or ViT).
They can snap in different unlearning methods (like "Bad Teacher" or "Selective Synaptic Dampening").
They can snap in different ways to measure success.
You don't have to rebuild the whole toolbox to add a new piece; you just register the new piece, and the system handles the rest.

3. The Three-Stage Race
For every experiment, SUPREME runs a three-step process:

Stage 1 (Training): The AI learns from the full library of data.
Stage 2 (Unlearning): The AI tries to "forget" specific data (either a whole category, like "all photos of Bill Gates," or a random 0.1% of photos). It also creates a "Gold Standard" model that was retrained from scratch without that data.
Stage 3 (Evaluation): The system compares the "Unlearned" AI against the "Gold Standard." It asks: "Did the AI actually forget the target? Did it keep its skills on the rest of the data? Is it still secure?"

What They Found (The Demo)

The authors tested this system on a dataset of celebrity faces (Pins Face Recognition). They asked the AI to forget specific people (like Hugh Jackman or Bill Gates) or a random handful of photos.

The Surprise: When they ran the tests just once, the results looked very different from when they ran them ten times. Some methods looked amazing in a single run but performed poorly when averaged over ten runs.
The Conclusion: This proves that running multiple seeds is essential. You cannot trust a single test result. The "luck of the draw" (randomness in how the AI starts) changes the outcome significantly.

Why This Matters

The paper doesn't claim to have invented a new way to unlearn data. Instead, it built the ruler to measure how well existing ways work.

Before SUPREME: Researchers were guessing which unlearning method was best based on single, potentially lucky experiments.
With SUPREME: Researchers can now run fair, large-scale comparisons to see which methods are truly robust and which ones are just getting lucky.

Summary

SUPREME is a new, open-source tool that lets researchers run AI "unlearning" tests faster and more fairly by using multiple computer chips at once. It showed that previous tests were often too small to be reliable, and that to truly know if an AI has "forgotten" something, you have to test it many times, not just once.

Note: The paper focuses entirely on the technical framework and the evaluation of methods on a specific dataset of celebrity faces. It does not discuss real-world applications like deleting user data from social media, nor does it make claims about clinical or legal uses.

Technical Summary: SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation

Problem Statement
Machine unlearning aims to remove the influence of specific training data from a trained model without retraining from scratch. While numerous approximate methods (e.g., fine-tuning, distillation, parameter-importance schemes) have been proposed, evaluating them presents significant challenges. Rigorous evaluation requires repeating the entire pipeline—training, unlearning, and assessment—across multiple random seeds to account for stochasticity in initial weights, data ordering, and unlearning operations. Existing image classification unlearning frameworks (e.g., MUBox, ERASURE, Deep Unlearn) typically operate on a single GPU with full precision. This limitation restricts the number of seeds that can be evaluated in a reasonable timeframe, potentially leading to misleading conclusions based on single-seed results. Furthermore, as model architectures (e.g., Vision Transformers) and datasets grow, the computational cost of these evaluations increases, necessitating more efficient infrastructure.

Methodology: The SUPREME Framework
The authors introduce SUPREME (Standardised Unlearning Platform for Reproducible Method Evaluation), an open-source framework designed to distribute the unlearning evaluation pipeline across multiple GPUs. The framework is built on PyTorch and Lightning Fabric, supporting distributed strategies including Distributed Data Parallel (DDP), Fully Sharded Data Parallel (FSDP), and DeepSpeed ZeRO.

The core of SUPREME is a seeded multi-stage pipeline (Algorithm 1) that executes three distinct stages across $P$ devices:

Stage 1 (Training): A model $M_o$ is trained on the full dataset $D$ using a specific training seed.
Stage 2 (Unlearning & Retraining): For each forget target, the framework defines a forget set ( $D_f$ $D_{f}$ ) and a retain set ( $D_r$ $D_{r}$ ). It then:
- Retrains a baseline model $M_r$ from scratch on $D_r$ using a specific unlearning seed.
- Applies an unlearning method $a$ to $M_o$ using $D_f$ and $D_r$ to produce the unlearned model $M_u$ .
- Both retraining and unlearning utilize gradient synchronization across devices.
Stage 3 (Evaluation): $M_u$ is evaluated against $M_r$ using configured metrics on test sets ( $D'_f$ and $D'_r$ ). Results are aggregated via an all-gather operation.

The framework supports two unlearning scenarios:

Targeted Unlearning: Removes all samples associated with a specific class label.
Random-Sample Unlearning: Removes a random subset of samples (defined by a percentage) drawn from the entire dataset.

Key Contributions
The paper outlines three primary contributions:

Registry-Based Extensibility: SUPREME employs a design where datasets, model architectures, unlearning methods, evaluation metrics, and scenarios are registered via module paths. New components can be added by implementing standard interfaces and registering the module path, without modifying the core framework code.
Multi-GPU Architecture: Unlike prior tools, SUPREME distributes training, unlearning, and evaluation across multiple accelerators. It supports various precision modes (e.g., mixed precision, 8-bit/4-bit via BitsandBytes) and distributed strategies, significantly reducing the time required for multi-seed evaluations.
Demonstration on Pins Face Recognition: The authors demonstrate the framework using the Pins Face Recognition dataset with ResNet18 and Vision Transformer (ViT) architectures. They evaluate multiple unlearning methods under full-class and random-sample scenarios across ten seeds.

Experimental Results
The demonstration was conducted on a single NVIDIA L40S GPU to maintain numerical parity with reference implementations, though the framework supports multi-GPU execution. Key findings include:

Across-Seed Variance: The study highlights that single-seed results can be highly misleading. For instance, in the random-sample scenario, the accuracy difference on the forget set ( $\Delta Acc_{D'_f}$ ) for "Random Labels" on ResNet18 showed a standard deviation of $\pm 34.03$ percentage points across ten seeds. Similarly, "Selective Synaptic Dampening" on ViT showed a deviation of $\pm 37.99$ . This confirms that multi-seed evaluation is necessary for consistent comparisons.
Metric Sensitivity: In the full-class scenario, Membership Inference Attack (MIA) differences ( $\Delta MIA$ ) were minimal across all methods (mostly within $\pm 0.05$ ), failing to distinguish between methods. Conversely, forget-set accuracy differences ( $\Delta Acc_{D'_f}$ ) varied significantly (from 0.00 to 89.44 on ResNet18), proving to be a more discriminative metric in this setting.
Performance: The framework successfully executed the full pipeline for both ResNet18 and ViT, capturing the variance introduced by different seeds and demonstrating the feasibility of large-scale, reproducible evaluation.

Significance and Claims
The authors position SUPREME as a tool to enable consistent, reproducible comparisons of unlearning methods, addressing the growing complexity of models and the necessity of multi-seed evaluation. They explicitly state that the reported numbers characterize the framework's capabilities rather than providing a definitive ranking of the unlearning methods, as the demonstration was limited to a single dataset, two architectures, and a single device. The paper concludes that future work will extend the demonstration to other registered components and utilize independent seeds for training, unlearning, and evaluation stages to decompose the sources of variance further. The framework is intended to facilitate the rigorous benchmarking required as the field of machine unlearning matures.

The Problem: The "One-Seed" Trap

The Solution: SUPREME

What They Found (The Demo)

Why This Matters

Summary

More like this