🤖 AI

EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure

The paper introduces EASE, a federated multimodal unlearning framework that effectively removes forgotten data by severing cross-modal reconstruction channels and isolating client-specific update directions through an entanglement-aware design involving anchor closure, subspace decomposition, and a direction-selective forget lock.

Original authors: Zihao Ding, Beining Wu, Jun Huang

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihao Ding, Beining Wu, Jun Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Right to be Forgotten" in a Group Chat

Imagine a group of friends (the clients) who want to build a super-smart AI together to recognize animals and describe them. They don't want to share their private photos with a central boss (the server); they only share their "study notes" (mathematical updates). This is Federated Multimodal Learning.

But what happens if one friend says, "I want to leave the group and have my photos completely erased from the AI's memory"? This is Federated Unlearning.

The problem is that in a multimodal AI (which sees both images and reads text), the knowledge is deeply tangled. If you just delete the notes about "dogs," the AI might still remember dogs because the text descriptions are still there, or because the math used by other friends is so mixed up that you can't tell which part belongs to the person who left.

The authors of this paper, EASE, found that previous methods failed because they didn't untangle these knots properly. They left three specific "anchors" (hidden hooks) that kept the forgotten knowledge alive. EASE is a new framework designed to cut all three hooks.

The Three "Anchors" (Why Previous Methods Failed)

The authors discovered that when you try to delete a client's data, three things keep the memory alive, acting like anchors holding a boat in place:

1. The Modality Anchor (The "One-Sided Eraser" Problem)

The Analogy: Imagine a couple writing a story together. One person writes the pictures, the other writes the words. They are so in sync that if you erase the pictures, the words still describe the pictures perfectly. If you try to "unlearn" the pictures by only changing the picture-writer's notes, the word-writer's notes act as a backup anchor, pulling the picture-writer back to the original story.
The Fix (Bilateral Knowledge Excision): EASE doesn't just erase the pictures; it erases the pictures and the words simultaneously. It forces both sides to let go of the specific pairing at the same time, so neither side can pull the memory back.

2. The Unique-Subspace Anchor (The "Shared Notes" Problem)

The Analogy: Imagine the group is studying for a math test. One student (the one leaving) has a unique way of solving a specific problem (a "unique direction"). However, their notes are mixed with notes from other students who are staying. If you try to delete the leaving student's notes by just throwing away a whole chunk of the textbook, you accidentally delete the unique math tricks the staying students need.
The Fix (Gradient Subspace Decomposition): EASE acts like a super-smart librarian. It looks at the math notes and separates them into two piles:
1. The "Unique" pile: Things only the leaving student knew. (These get deleted).
2. The "Shared" pile: Things everyone learned together. (These are kept safe).
  It surgically removes only the unique pile without touching the shared knowledge.

3. The Temporal Re-anchoring (The "Slippery Slope" Problem)

The Analogy: Imagine you successfully delete the leaving student's notes. But the next day, the group starts studying again. Because the remaining students' notes still have a tiny, hidden bias toward the leaving student's style, they accidentally start rewriting the deleted notes into the group's new study guide. The memory creeps back in over time.
The Fix (Forget Lock): EASE puts a "lock" on the specific directions where the deleted knowledge used to live. As the group continues to study, the AI is mathematically forced to stay away from those specific directions. It's like putting a "Do Not Enter" sign on the deleted memory lane so the group can't accidentally wander back in.

How EASE Works (The Solution)

The authors built a system called EASE (Entanglement-Aware Subspace Excision) that combines these three fixes into one workflow:

Simultaneous Erasure: It cuts the connection between images and text on both sides at once (closing the Modality Anchor).
Surgical Separation: It uses a mathematical technique (Cosine-Sine decomposition) to find exactly which parts of the math belong only to the person leaving and which parts belong to the group (closing the Unique-Subspace Anchor).
The Lock: It applies a "Forget Lock" that prevents the AI from drifting back into the deleted memory lane during future training sessions (closing the Temporal Re-anchoring).

The Results

The paper tested EASE on real-world datasets (like Flickr30K and COCO) where the AI learns to match images with text descriptions.

The Goal: The AI should behave exactly as if the leaving client's data was never there (like a "retrain" from scratch), but without the massive cost of retraining everything.
The Outcome: EASE was significantly better than previous methods.
- Forget Side: It successfully erased the specific data (e.g., if a client left, the AI stopped recognizing their specific dog photos).
- Retain Side: It kept the AI smart for everyone else (it didn't forget how to recognize cats or other dogs).
- Efficiency: It did this much faster and cheaper than retraining the whole model from scratch.

Summary

Think of EASE as a precision eraser for group AI projects. Instead of scrubbing the whole whiteboard and risking a mess, it identifies the exact sticky notes that need to be removed, peels them off without smudging the notes underneath, and puts up a barrier so no one accidentally sticks them back on later. This ensures the "Right to be Forgotten" is actually respected in complex, multi-sensory AI systems.

1. Problem Statement

The paper addresses the challenge of Federated Multimodal Unlearning (FML) in the context of large-scale vision-language models (e.g., CLIP). In FML, multiple clients collaboratively train a model on private image-text pairs without sharing raw data. When a client withdraws, or specific data (samples/classes) must be removed (e.g., due to privacy rights or harmful content), the system must "unlearn" this data without retraining from scratch.

The authors identify that existing federated unlearning methods fail in multimodal settings because they do not account for three specific residual anchors that allow forgotten knowledge to persist:

Modality Anchor: Due to bilinear coupling in contrastive losses (e.g., InfoNCE), the gradient of one modality (e.g., text) depends on the embeddings of the other (e.g., image). If only one branch is modified, the untouched branch acts as an anchor, reconstructing the forgotten alignment.
Unique-Subspace Anchor: Client gradient subspaces are entangled. Forget-specific directions are mixed with directions shared by retained clients. Existing methods treat the forget subspace as a monolithic block, leading to a trade-off: either they fail to remove the data (under-erasure) or they damage retained knowledge (over-erasure).
Temporal Re-anchoring: Even if anchors are closed at the moment of unlearning, subsequent Federated Averaging (FedAvg) rounds with retained clients can gradually rewrite the removed directions, causing the model to "remember" the forgotten data over time.

2. Methodology: EASE Framework

The authors propose EASE (Entanglement-Aware Subspace Excision), a framework designed to close all three anchor channels simultaneously. It operates on top of a frozen multimodal backbone using lightweight LoRA adapters.

A. Bilateral Knowledge Excision (BKE) – Closing the Modality Anchor

Mechanism: Instead of modifying only the visual or textual branch, EASE displaces both branches simultaneously.
Logic: In a bilinear similarity model, the alignment gradient depends on the product of both embeddings. By displacing both $z_v$ and $z_t$ away from their original values, the bilinear similarity can no longer reconstruct the forgotten pairing.
Outcome: This prevents the "untouched" modality from acting as a memory anchor that pulls the modified branch back to the original alignment.

B. Gradient Subspace Decomposition (GSD) – Closing the Unique-Subspace Anchor

Mechanism: The server collects historical gradient updates from clients and performs Singular Value Decomposition (SVD) to extract the forget subspace ( $S_f$ ) and retain subspace ( $S_r$ ).
Logic: Using Cosine-Sine decomposition, the method calculates the principal angles between $S_f$ $S_{f}$ and $S_r$ $S_{r}$ .
- Directions nearly orthogonal to $S_r$ are identified as Unique-Subspace Anchors (removable).
- Directions highly aligned with $S_r$ are identified as Retain Support (must be preserved).
Action: The server projects the global model parameters onto the complement of the unique subspace, effectively excising only the forget-exclusive directions while preserving the shared support.

C. Projection with Forget Lock (PFL) – Closing Temporal Re-anchoring

Mechanism: A two-pronged approach to prevent drift during continued training:
1. Server-side Projection: Immediately after unlearning, the global model is projected to remove unique-subspace components.
2. Client-side Forget Lock: During subsequent local SGD rounds, clients optimize a combined objective: $L_{total} = L_{align} + \alpha \cdot L_f$ .
Logic: $L_f$ is a quadratic penalty that specifically penalizes drift along the identified unique subspace directions ( $U_\mu$ ).
Outcome: This creates a "lock" that bounds the drift of forgotten knowledge back into the model, while leaving the orthogonal complement (retained knowledge) unconstrained for recovery.

3. Key Contributions

The Anchor Principle: The paper formally identifies and characterizes three distinct mechanisms (Modality, Unique-Subspace, and Temporal Re-anchoring) that cause failure in federated multimodal unlearning. It demonstrates that closing any subset of these leaves a measurable reconstruction path.
EASE Framework: A unified solution combining BKE, GSD, and PFL. It is the first method to address cross-modal coupling and subspace entanglement simultaneously in a federated setting.
Theoretical Guarantees: The authors provide geometric proofs showing that:
- BKE eliminates the dominant modality anchor signal.
- GSD bounds the collateral damage to retained knowledge based on the threshold $\delta$ .
- PFL bounds the temporal drift of forgotten knowledge inversely proportional to the lock strength $\alpha$ .
Comprehensive Evaluation: Extensive experiments on Flickr30K, MS COCO, and TextCaps using CLIP-B/32, CLIP-L/14, and SigLIP backbones.

4. Experimental Results

EASE was compared against nine baselines (including FedEraser, Ferrari, FedOSD, and GradAscent) across three unlearning scenarios: Client, Class, and Sample.

Unlearning Completeness: EASE significantly outperforms all baselines in reducing retrieval performance on the forget set (Forget R@1). For example, on Flickr30K with CLIP-B/32, EASE achieved a Forget R@1 of 0.3 (near the retrain reference of 0.1), whereas baselines like Ferrari and FedOSD remained high (>60-80%), indicating they failed to forget.
Retention Integrity: Unlike aggressive methods (e.g., GradAscent) that destroy retained knowledge, EASE maintains high performance on the retain set. It matched the retrain reference within 4.2 R@1 points on the retain side for client unlearning, whereas other methods suffered significant drops.
Privacy Verification: EASE showed the lowest Membership Inference Attack (MIA) scores and Likelihood-Ratio Attack (LiRA) TPR, confirming that forgotten data is indistinguishable from non-members.
Ablation Studies: Removing any single component (BKE, GSD, or Lock) resulted in distinct failure modes, validating the necessity of addressing all three anchors.
Efficiency: EASE requires significantly less communication cost (approx. 20-25% of full retraining) while achieving performance comparable to a full retrain.

5. Significance

Paradigm Shift: The paper moves federated unlearning from a "black-box erasure" approach to a "geometric, entanglement-aware" approach. It highlights that in multimodal systems, unlearning cannot be done unilaterally on one modality or by treating gradients as independent blocks.
Practical Applicability: By using LoRA adapters and projection operators, EASE is computationally efficient and compatible with existing federated learning infrastructure, making it viable for real-world deployment in regulated domains (e.g., healthcare, finance) where the "Right to be Forgotten" is critical.
Robustness: The framework ensures that unlearning is durable against continued training, solving the "drift" problem that plagues many existing unlearning algorithms.

In conclusion, EASE provides a theoretically grounded and empirically superior solution for removing data from federated multimodal models, effectively closing the reconstruction channels that previously hindered true unlearning.