🤖 machine learning

Rethinking Federated Unlearning via the Lens of Memorization

This paper proposes Federated Memorization Pruning (FedMemPrune), a novel federated unlearning approach that leverages a new Grouped Memorization Evaluation metric to distinguish and remove unique memorized information while preserving overlapping patterns, thereby achieving effective unlearning without compromising model utility.

Original authors: Jiaheng Wei, Yanjun Zhang, He Zhang, Leo Yu Zhang, Chao Chen, Kok-Leong Ong, Jun Zhang, Yang Xiang

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaheng Wei, Yanjun Zhang, He Zhang, Leo Yu Zhang, Chao Chen, Kok-Leong Ong, Jun Zhang, Yang Xiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Right to be Forgotten" in a Group Chat

Imagine a group of friends (clients) who want to build a single, super-smart study guide (a machine learning model) together. They can't share their private notes because they are too personal, so instead, they only share their "aha!" moments and summaries (model updates) with a central teacher (the server). This is Federated Learning.

But what happens if one friend, let's call them "Alex," decides to quit the group and demands that the study guide forget everything Alex ever contributed? This is Federated Unlearning.

The law (like GDPR) says Alex has a "Right to be Forgotten." The easy way to do this is to throw away the whole study guide and start from scratch with the remaining friends. But that takes forever and costs a lot of energy. The paper argues that the current "quick fix" methods are actually making mistakes.

The Problem: Throwing the Baby Out with the Bathwater

The authors argue that existing methods are too aggressive. They try to erase everything related to Alex, even the things Alex learned that were actually shared by everyone else.

The Analogy:
Imagine the group is learning to bake a cake.

Shared Knowledge: Everyone agrees that you need flour and eggs. This is general knowledge.
Alex's Unique Memory: Alex has a specific, weird trick for cracking eggs that no one else uses. This is "memorization."

If Alex leaves, the current methods try to remove all knowledge about eggs, even the fact that you need them. This hurts the remaining friends because they still need eggs to bake the cake. The model becomes worse at baking for everyone else.

The paper asks: Should we remove the shared knowledge just because Alex is gone? The answer is No. We should only remove Alex's unique tricks (memorization) while keeping the shared rules (overlapping information).

The Solution: "Memorization Pruning" (FedMemPrune)

The authors propose a new method called FedMemPrune. Think of it as a smart editor for the study guide.

Instead of erasing whole pages, the editor looks for the specific sentences that only Alex wrote and deletes only those.

How it works (The Three Steps):

Finding the "Redundant" Notes:
The system looks at the remaining friends' notes. It asks: "Which parts of the study guide are not being used or updated by the remaining friends?"
- Analogy: If the remaining friends are all writing about "flour," but one paragraph is only about "Alex's weird egg trick," that paragraph is "redundant" to the group. It's likely where Alex's unique memory lives.
Resetting the Memory:
The system takes those specific "redundant" parts of the study guide and scrubs them clean (re-initializes them). It doesn't just delete them; it resets them to a blank state, like erasing a whiteboard.
Fine-Tuning:
The remaining friends quickly review the guide together to make sure the "flour" and "eggs" sections are still perfect. Because the shared knowledge was never touched, the guide remains excellent for everyone else.

Why This is Better

The paper tested this against other methods and found three major wins:

Better "Forgetting": It actually removes Alex's unique tricks better than other methods.
Better "Remembering": It keeps the group's shared knowledge intact, so the study guide doesn't get worse for the other friends.
Fairness: It treats the remaining friends equally. Some methods accidentally hurt the friends who had the most similar notes to Alex, but this method protects them.
Speed: It is much faster than starting from scratch (retraining), saving about 50% of the time.

The "Grouped Memorization" Scorecard

The authors also realized that we need a better way to check if the forgetting worked. You can't just look at the final grade; you need to check specific questions.

They created a new test called Grouped Memorization Evaluation.

They take the questions Alex was asked and sort them by how "unique" or "memorized" they were.
They check if the new study guide fails on the unique questions (good!) but still passes the common questions (also good!).
This proves that the system isn't just guessing; it's specifically forgetting the right things.

Summary

In short, this paper says: Don't be a brute when you need to forget.

If you want to remove a person's contribution from a group project, don't burn the whole project down. Instead, identify the specific, unique parts that only that person knew, erase those, and let the rest of the team polish up the shared parts. This keeps the project strong, fair, and fast.

Technical Summary: Rethinking Federated Unlearning via the Lens of Memorization

Problem Statement
Federated Learning (FL) is increasingly required to support machine unlearning to comply with privacy regulations such as GDPR and CCPA (the "right to be forgotten"). While retraining a model from scratch after removing a client's data is the gold standard for unlearning, it incurs prohibitive computational and communication costs. Existing federated unlearning (FU) approaches attempt to approximate this retraining by perturbing representations, utilizing historical updates, or applying gradient ascent. However, the authors argue that these methods often overlook the critical distinction between overlapping information (shared generalizable patterns between the unlearning client and remaining clients) and unique memorization information (client-specific details). Current algorithms frequently attempt to eliminate the influence of the entire unlearning dataset, inadvertently removing overlapping patterns. This leads to ineffective unlearning (failing to remove specific memorized data) and unfairness (disproportionately degrading the performance of remaining clients who share similar patterns with the unlearning client).

Methodology
The paper proposes a framework grounded in information theory and the concept of memorization, leading to the development of Federated Memorization Pruning (FedMemPrune).

Theoretical Foundation (Information Decomposition):
The authors formalize unlearning using Partial Information Decomposition. They define the information learned by a model from the union of unlearning data ( $D_u$ ) and remaining data ( $D_r$ ) as comprising:
- Synergistic Information: Jointly provided by $D_u$ and $D_r$ .
- Redundant/Overlapping Information ( $F_g$ ): Shared between $D_u$ and $D_r$ .
- Unique/Memorization Information ( $F_m$ ): Specific to $D_u$ .
The authors argue that the synergistic component is negligible in general unlearning scenarios. Therefore, the goal of unlearning should not be to erase all information from $D_u$ , but specifically to remove the unique memorization information ( $F_m$ ) while preserving the overlapping information ( $F_g$ ). Removing $F_g$ harms generalization and fairness, whereas removing $F_m$ achieves the desired "forgetting" without sacrificing utility.
Evaluation Metric: Grouped Memorization Evaluation (GME):
To address the lack of fine-grained evaluation metrics, the authors introduce GME. This metric calculates a memorization score for each example in the unlearning dataset, defined as the difference in correct classification probability between a model trained on $D_u \cup D_r$ and a model trained only on $D_r$ . Examples are then grouped by these scores (e.g., high-memorization vs. low-memorization). Unlearning efficacy is measured by how closely the unlearned model's performance on these subgroups matches the retrained baseline, specifically targeting the removal of high-memorization examples.
Algorithm: FedMemPrune:
FedMemPrune is a three-stage pruning-based approach:
- Stage 1 (Locating Memorization Parameters): The algorithm identifies redundant parameters in the global model. These are defined as parameters that receive minimal gradient updates from the remaining clients ( $D_r$ ). The hypothesis is that parameters critical to $D_r$ encode overlapping/generalizable knowledge, while redundant parameters encode the unique memorization specific to $D_u$ .
- Stage 2 (Resetting): The identified redundant parameters are re-initialized using the original initialization strategy (e.g., Kaiming Uniform), effectively erasing the memorized information.
- Stage 3 (Fine-tuning): The model is fine-tuned on the remaining clients ( $D_r$ ) to restore generalization performance and reconstruct the overlapping representations.

Key Contributions

Conceptual Shift: The paper redefines federated unlearning as a process of removing client-specific memorization rather than erasing all data influence, arguing that overlapping information should be preserved to maintain model utility and fairness.
Novel Metric: The introduction of Grouped Memorization Evaluation (GME), which enables example-level assessment of unlearning efficacy, distinguishing between the forgetting of memorized tail data and the preservation of generalizable head data.
Proposed Algorithm: The development of FedMemPrune, which leverages parameter redundancy relative to remaining data to selectively prune memorization information.

Experimental Results
Extensive experiments were conducted on CIFAR-10, CIFAR-100, and EMNIST under both IID and Non-IID settings, comparing FedMemPrune against baselines such as Halimi et al. (gradient ascent), Liu et al. (Fisher Information Matrix), FedRecovery, FedAU, FedOSD, and NoT.

Unlearning Efficacy: FedMemPrune closely matches the performance of the retrained baseline, particularly within high-memorization subgroups. For instance, on CIFAR-100 (Non-IID), the method achieved classification accuracies on high-memorization groups (e.g., 27.25% vs. 21.74% for the baseline) that were significantly closer to the retrained model than other baselines, which often failed to forget high-memorization data or over-forgot.
Generalization Performance: By preserving overlapping information, FedMemPrune maintains high test accuracy on the retained dataset, often outperforming other unlearning baselines and approaching the retrained baseline (e.g., 63.33% on CIFAR-100 IID).
Fairness: The method demonstrates superior local fairness, minimizing the variance in performance changes across remaining clients. This confirms that preserving overlapping information prevents disproportionate degradation for clients sharing data with the unlearning client.
Efficiency: FedMemPrune achieves the target performance in approximately 50% less time compared to full retraining, as it avoids the need to retrain the entire model from scratch.

Significance and Claims
The authors claim that their work provides both theoretical insights and practical strategies for strengthening federated unlearning. By distinguishing between memorization and overlapping information, they argue that effective unlearning should target only the former. The paper asserts that FedMemPrune successfully achieves robust and efficient unlearning without sacrificing the utility of retained knowledge or fairness among clients. The work highlights that current evaluation protocols are insufficient and that a memorization-aware perspective is necessary to rigorously assess and improve federated unlearning mechanisms. The authors emphasize that their approach is particularly suitable for scenarios where the unlearning dataset is unavailable during the unlearning phase, as it relies on the gradients of the remaining clients to identify what to prune.