🤖 machine learning

Forgettable Federated Linear Learning with Certified Data Unlearning

This paper introduces Forgettable Federated Linear Learning, a framework that approximates deep neural networks with linear models to enable certified, efficient, and secure federated unlearning without requiring additional client communication or historical model storage.

Original authors: Ruinan Jin, Minghui Chen, Qiong Zhang, Xiaoxiao Li

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruinan Jin, Minghui Chen, Qiong Zhang, Xiaoxiao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you and a group of friends are building a giant, collaborative puzzle to solve a mystery. This is Federated Learning (FL). Instead of everyone sending their puzzle pieces to a central table (which would be a privacy nightmare), everyone keeps their pieces at home. They just send the instructions on how to fit their pieces together to a central leader. The leader combines these instructions to build a better picture, then sends the updated instructions back. Everyone learns together without ever showing their private pieces.

But what happens if one friend brings a poisoned piece? Maybe they secretly painted a hidden symbol on their piece that makes the whole puzzle look wrong when that symbol appears. Or, maybe that friend just wants to leave the group and have their piece completely erased from the final picture (the "Right to be Forgotten").

This is where Federated Unlearning (FU) comes in. It's the process of removing a friend's contribution without having to throw away the whole puzzle and start over.

The Problem with Current Methods

Currently, trying to remove a friend's piece is a nightmare:

The "Do-Over" Method: You could ask everyone to rebuild the puzzle from scratch without that one friend. This takes forever and costs a lot of energy (computing power).
The "Ask the Friend" Method: Some methods ask the friend who wants to leave to do some extra math on their own computer and send it back. But what if that friend is the one who poisoned the puzzle? They might lie or refuse to cooperate.
The "Hoarding" Method: Some methods require the leader to save a copy of every single version of the puzzle as it was built, step-by-step. This fills up the storage room instantly.

The New Solution: F2L2

The authors of this paper propose a clever new way called Forgettable Federated Linear Learning (F2L2). They use two main tricks to make unlearning easy, fast, and safe.

Trick 1: The "Linear Sketch" (Federated Linear Training)

Deep neural networks (the complex AI models used today) are like incredibly tangled knots. Untangling one specific thread (removing one person's data) is nearly impossible without undoing the whole knot.

The authors' first trick is to stop trying to untangle the knot. Instead, they use a pre-trained model (a model already trained on public data) as a starting point. They then treat the complex model like a straight line (a "linear approximation") around that starting point.

The Analogy: Imagine you are drawing a complex mountain range. It's hard to erase just one hill. But if you zoom in very close to the peak, the mountain looks like a simple, straight ramp. If you want to remove a specific part of that ramp, you can just calculate a simple math adjustment. You don't need to redraw the whole mountain; you just adjust the angle of the ramp.

This "linear ramp" is easy to work with mathematically. It turns a messy, complex problem into a simple one where you can just do a quick calculation to remove someone's data.

Trick 2: The "Magic Eraser" (FedRemoval)

Once the model is treated as a simple "ramp," the server (the leader) can perform a Newton Step.

The Analogy: Think of the model as a ball sitting in a valley. The ball's position represents the final model. If you want to remove a friend's data, you don't need to ask them to push the ball or look at old photos of the ball. You just need to know how hard they pushed it in the first place (their final gradient, which they already sent during training).
The server takes that "push" information and calculates exactly how to move the ball back to where it would have been if that friend had never pushed it. It's like a magic eraser that works instantly on the server side.

Why This is a Big Deal

The paper claims this new method, F2L2, solves the three biggest headaches of current unlearning:

No "Ask the Friend" Needed: The server does all the work. It doesn't need to contact the person leaving, even if they are malicious or offline.
No "Hoarding" Needed: The server doesn't need to save every version of the model. It only needs the final math from the training. This saves massive amounts of memory.
It's Certified: The authors didn't just guess this works; they proved it with math. They showed that the result of their "Magic Eraser" is almost identical to what you would get if you had thrown away the whole puzzle and rebuilt it from scratch without that friend.

The Results

The team tested this on everything from simple number recognition (MNIST) to complex image recognition (like identifying flowers or cars). They even tested it on "Foundation Models" (the huge, powerful AI models everyone is talking about right now).

The Outcome: In every test, F2L2 successfully removed the "poisoned" data (the backdoor attacks) so that the model stopped misbehaving.
The Trade-off: Crucially, removing the bad data didn't ruin the model's ability to do its job. The model remained just as smart as before, but without the bad influence.

Summary

In short, this paper introduces a way to unlearn in a group AI setting that is:

Fast: No need to retrain the whole model.
Secure: Doesn't rely on the person leaving to cooperate.
Efficient: Doesn't require saving terabytes of old data.
Proven: Mathematically guaranteed to work almost as well as starting over.

It turns the impossible task of "erasing a specific memory from a complex AI" into a simple, one-step calculation on the server.

Technical Summary: Forgettable Federated Linear Learning with Certified Data Unlearning (F2L2)

Problem Statement
Federated Learning (FL) enables collaborative model training while preserving data privacy, yet it faces significant challenges regarding the "right to be forgotten" and the removal of malicious contributions (e.g., poisoned data from backdoor attacks). Existing Federated Unlearning (FU) methods suffer from critical limitations that hinder real-world adoption:

Client Dependence: Many methods require the cooperation of either retained clients (for retraining or fine-tuning) or target clients (for gradient ascent or customized optimization), introducing communication overhead and security risks if clients are malicious or unavailable.
Storage Inefficiency: Approaches like FedEraser require storing all historical global models, incurring prohibitive memory costs for large-scale Deep Neural Networks (DNNs).
Lack of Certification: Due to the nonlinearity of DNNs, most existing FU methods lack theoretical guarantees that the unlearned model is indistinguishable from one retrained from scratch.

The core research question addressed is: How can we design a practical FU framework that ensures certified unlearning without requiring retained client cooperation, target client participation, or historical model storage?

Methodology: The F2L2 Framework
The authors propose Forgettable Federated Linear Learning (F2L2), a framework that integrates a novel training pipeline with a server-side unlearning mechanism.

Federated Linear Training (FLT):
- Linearization: Inspired by the Neural Tangent Kernel (NTK) and its extensions to pre-trained models, FLT approximates a DNN linearly around a pre-trained weight vector $w_p^*$ . The model function is approximated as $f(x; w) \approx f(x; w_p^*) + \langle \nabla_w f(x; w_p^*), w - w_p^* \rangle$ .
- Quadratic Loss: This linearization transforms the loss function into a quadratic form (Mean Squared Error), ensuring convexity.
- Training Process: The server initializes the model using a pre-trained foundation model (FM) or a model trained on a small, trusted public dataset ( $D_p$ ). Clients then perform standard Federated Averaging (FedAvg) updates on this linearized model. Crucially, the pre-trained encoder parameters remain fixed (in the case of FMs), and only the linear layer weights are updated.
FedRemoval (Certified Unlearning):
- Newton Step: Because the loss is quadratic, the optimal weights after removing a target client's data ( $D_c$ ) can be computed via a closed-form Newton step: $w^- = w^* - [\nabla^2 L(w^*; D_-)]^{-1} \nabla L(w^*; D_-)$ .
- Server-Side Execution: To avoid the computational cost of inverting the Hessian over the distributed remaining dataset ( $D_-$ ), FedRemoval approximates the Hessian using the pre-trained dataset $D_p$ (which is available on the server).
- Efficiency: The unlearning process requires only the final gradients from the target client (already shared during training) and the server's local data. It involves zero additional communication with clients and zero storage of historical models. The update is computed entirely on the server.

Key Contributions

Novel Framework: F2L2 is the first federated unlearning solution combining distributed mixed-linear training (FLT) with a purely server-side unlearning mechanism (FedRemoval). It eliminates the need for client cooperation during unlearning and historical model storage.
Certified Guarantees: The authors derive a theoretical upper bound on the $L_2$ -norm discrepancy between the weights obtained via FedRemoval and those obtained by full retraining. This provides a rigorous certification that the unlearning process is effective.
Comprehensive Evaluation: Extensive experiments were conducted on six datasets (MNIST, FashionMNIST, CIFAR10, ImageNet, DomainNet, Flowers) and four model architectures, including foundation models (CLIP, BLIP2) using linear probing.

Experimental Results

Unlearning Efficacy: F2L2 successfully removes backdoor attacks, reducing the Backdoor Success Rate (BSR) to levels comparable to random guessing (e.g., ~5-10% on various datasets), while maintaining Test Accuracy (TA) close to the retrained ground truth.
Comparison to Baselines: F2L2 outperforms existing baselines (FedEraser, EWSGA, SGA, Flipping, CF, etc.) by achieving a better trade-off between TA and BSR. Unlike baselines that often require extra communication or storage, F2L2 achieves near-ground-truth performance with zero additional overhead.
Foundation Models: The method is effective when applied to large-scale foundation models (e.g., CLIP on ImageNet/DomainNet), maintaining high TA (~90-95%) while effectively unlearning poisoned clients.
Scalability and Stability: The method remains stable across varying numbers of clients (5 to 20) and different data heterogeneity settings (Non-IID distributions), though BSR increases slightly with higher heterogeneity, remaining near random guessing levels.
Efficiency: The Hessian approximation significantly reduces computational costs (FLOPs) and eliminates communication overhead during the unlearning phase.

Significance and Claims
The paper claims that F2L2 represents a promising pipeline for efficient and trustworthy Federated Unlearning. By leveraging the mathematical properties of linearized models around pre-trained weights, the authors demonstrate that it is possible to achieve certified unlearning without the traditional trade-offs of communication overhead, client cooperation, or massive storage requirements. The work suggests that linear approximations of DNNs, particularly in the context of foundation models, offer a viable path to scalable and secure federated systems that can respect user privacy rights and mitigate security threats like backdoor attacks.

The authors note that while the linear approximation is effective, extending the framework to broader FL settings beyond classification, generative tasks, or highly nonlinear regimes remains a direction for future research.