🤖 machine learning

Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks

This survey presents a unified, system-oriented taxonomy of multimodal unlearning methods, datasets, and benchmarks across vision, language, audio, and video modalities, addressing the challenge of selectively removing sensitive or copyrighted information from foundation models while preserving their overall utility.

Original authors: Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu, Vaidehi Patil

Published 2026-07-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu, Vaidehi Patil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart, super-creative robot friend who can draw pictures, write stories, sing songs, and make videos all at once. This robot learned by reading the entire internet, but in doing so, it accidentally memorized some things it shouldn't have: maybe a private photo of a neighbor, a copyrighted painting by a famous artist, or a weird, unsafe joke.

Now, the neighbor wants that photo forgotten, the artist wants their style erased, and the internet wants that joke gone. The old way to fix this was to delete the bad data and rebuild the robot from scratch. But that's like tearing down a skyscraper just to remove one cracked brick—it takes forever, costs a fortune, and is totally impractical.

This paper is a massive survey (a giant map of the current landscape) that explores a new, smarter way to fix this: Multimodal Unlearning. Instead of rebuilding the whole robot, these methods try to perform "selective forgetting," surgically removing just the bad memories while keeping the robot's ability to draw, write, and sing perfectly intact.

The Big Map: Where Do We Cut?

The authors organize all the different methods into a "system-first" map. Instead of just looking at the math, they look at when and where you intervene in the robot's brain. They found five main places to perform this surgery:

The Data Side (Before the Robot Learns): Imagine putting a "do not learn" sticker on the bad photos or text before the robot even sees them. Some methods tweak the data slightly so the robot can't memorize it, or they clean up the training library to remove "poisoned" pairs of images and words.
Training-Time Edits (While the Robot is Learning): This is like teaching the robot a new rule while it's studying. You tell it, "Hey, when you see this specific thing, don't remember it, but keep remembering everything else." The robot adjusts its brain weights to forget the target while holding onto the rest.
Architecture Constraints (Changing the Robot's Hardware): Sometimes, instead of changing the rules, you change the robot's structure. You might "freeze" (lock) certain parts of its brain so they can't change, or prune (cut away) specific connections that hold the bad memory, then let the robot relearn the good stuff around the gap.
Training-Free Unlearning (The Magic Eraser): This is the coolest part. These methods don't retrain the robot at all. They use math to directly edit the robot's brain weights or its internal representations (the way it "thinks" about things) in one quick step. It's like using a specific chemical solvent to dissolve just the bad ink without touching the rest of the page.
Decoding Time (During the Show): Imagine the robot is drawing a picture. At the very last second, before the image appears, a filter steps in and says, "No, don't draw that specific face." The robot's brain stays the same, but the output is steered away from the forbidden content.

The Rules of the Game

The paper is very clear about what these methods are trying to achieve and what they are not doing.

The Goal: The main goal is selective removal. You want to forget a specific "forget set" (like a private face or a copyrighted style) while keeping the "retain set" (everything else) working perfectly. The paper defines this mathematically: the robot after forgetting should act almost exactly like a robot that was never trained on the bad data in the first place.
What They Rule Out: The paper explicitly argues against the idea that you can just delete data and expect the robot to automatically forget. It shows that knowledge is distributed across the robot's brain; deleting the source file doesn't erase the memory. It also rules out the idea that current methods are perfect. They are often heuristic (based on trial and error and smart guesses) rather than having iron-clad mathematical proofs that the memory is 100% gone forever.
The "Unlearning" vs. "Hiding" Trap: The paper suggests that some methods might just be hiding the bad info rather than truly deleting it. If you poke the robot with the right "adversarial" questions (like a jailbreak), the forgotten memory might pop back out. The paper notes that while we have many methods, we don't yet have a guarantee that the memory is truly erased, especially against clever attacks.

The Toolkit: Datasets and Benchmarks

To test these methods, researchers use specific "training grounds." The paper lists a bunch of datasets used for this:

For Faces: They use datasets like CelebA (202,599 images) and VGGFace2 (3.3 million face images) to test if the robot can forget a specific person's face.
For Copyright: They use UnlearnCanvas (60 artistic styles) to see if the robot can forget a specific artist's style.
For Safety: They use datasets like I2P (4,700 prompts) to test if the robot can forget how to generate unsafe or inappropriate content.
For Audio: They use VoxCeleb1 (150,000 utterances) to test if the robot can forget a specific speaker's voice.

The paper highlights that we have benchmarks (like MU-Bench and MLLMU-Bench) to measure how well these methods work. These benchmarks check three things:

Did it forget? (Does the robot fail to recognize the bad thing?)
Did it keep the good stuff? (Is the robot still good at drawing and writing about everything else?)
Is it safe? (Can a hacker trick the robot into remembering the bad thing again?)

The Reality Check: What's Still Broken?

The authors are honest about the gaps. They suggest that while we have many cool tools, we are still in the "early days."

No Magic Guarantees: Most methods are simulated or tested on specific models. We don't have a universal mathematical proof that says, "This method will always delete the memory forever, no matter what."
The "Resurfacing" Problem: The paper suggests that in generative models (like image creators), forgotten concepts might resurface if the robot is fine-tuned later or if someone uses a clever prompt. It's like forgetting a song, but then hearing a similar tune and remembering it again.
The Trade-off: There is often a trade-off. The more aggressively you try to forget, the more you might hurt the robot's ability to do other things. The paper notes that deleting just a tiny bit of data can sometimes cause a measurable drop in performance on other tasks.
Cross-Modality Leaks: If you teach the robot to forget a face in images, does it also forget that face in text descriptions? The paper suggests this is a huge open challenge; safety in one mode (text) doesn't always mean safety in another (image).

The Bottom Line

This paper is a survey, meaning it's a comprehensive guide to what exists right now, not a single new invention. It organizes the chaos of "how to make AI forget" into a clear system. It suggests that we have powerful tools to perform "surgery" on AI models, but we are still figuring out how to make that surgery 100% reliable, safe from hackers, and perfect at not damaging the rest of the robot's brain.

The authors are releasing a curated repository (a public collection of code and data) to help others build on this. They are hopeful that by mapping out the problems—like the need for better benchmarks and stronger guarantees—we can eventually build AI that respects privacy and copyright without needing to be rebuilt from the ground up every time someone asks to be forgotten.

Technical Summary: Multimodal Unlearning Across Vision, Language, Video, and Audio

1. Problem Definition

The rapid adoption of Multimodal Foundation Models (MFMs)—including Vision-Language Models (VLMs), Diffusion Models (DMs), Large Language Models (LLMs), and Audio Foundation Models (AFMs)—has introduced significant risks regarding the inadvertent encoding of sensitive, copyrighted, biased, or unsafe cross-modal associations. These models, trained on web-scale data, often memorize specific instances (e.g., private faces, copyrighted artwork) or undesirable concepts (e.g., harmful tropes).

When faced with deletion requests (e.g., Right to be Forgotten) or policy updates, the standard solution of retraining from scratch is often computationally impractical. Furthermore, targeted forgetting is technically challenging because knowledge in MFMs is distributed across shared representations; removing an association in one modality (e.g., text) can inadvertently disrupt unrelated behaviors in another (e.g., image generation) or fail to fully erase the concept due to residual cross-modal leakage. The core problem is achieving selective removal of specific data or concepts across modalities while retaining overall utility and model performance on the remaining data.

2. Methodology and Taxonomy

This survey proposes a system-oriented taxonomy that organizes multimodal unlearning methods not by optimization objectives, but by intervention stage and control pathway. This framework covers the entire pipeline from data preparation to inference.

2.1 Formalization

The goal is defined as removing the influence of a forget set $D_f$ while preserving utility on retained data $D_r$ . The unlearning procedure $U$ applied to the original model $M_o$ and data $D$ should produce a model $M_u$ whose distribution is indistinguishable from a model retrained on $D \setminus D_f$ . This is formalized using an $(\epsilon, \delta)$ unlearning criterion, mirroring Differential Privacy, to provide theoretical guarantees on the equivalence between unlearning and retraining.

2.2 Intervention Stages

The survey categorizes methods into five primary intervention points:

Data-Side Interventions:
- Data-Path Perturbation: Edits inputs (images/text) rather than weights to reduce the learnability of targeted clusters (e.g., using unlearnable examples or cloaks) while preserving utility on the rest of the corpus.
- Data Hygiene & Prompt Normalization: Curates training data by down-weighting suspicious pairs or normalizing prompts to canonical forms to mitigate spurious associations before training begins.
Training-Time Edits:
- Direct Gradient: Formulates unlearning as targeted risk minimization, suppressing responses on the forget set while maintaining performance on the retain set. This includes objectives for VLMs and diffusion models that decouple cross-modal structures.
- Constrained Updates: Imposes bounds during optimization to limit residual competence on the forget set and restrict deviation from a reference model, balancing forgetting efficacy with model integrity.
- Mask-Driven Selective Unlearning: Restricts updates to localized parameters, features, or spatial regions identified via saliency or attention maps, minimizing collateral damage to retained behaviors.
- Distillation-Based: Transfers behavior from a "teacher" to a "student" model guided toward safe targets, often without direct access to original training data.
Architecture-Constrained Unlearning:
- Architecture Editing: Modifies the network structure directly (e.g., pruning, freezing, or regrowing specific layers/tensors) to block pathways encoding the forget signal.
- Layer-Scoped Constrained Updates: Identifies the specific layers where a concept concentrates and restricts edits to those layers to limit parameter drift.
Training-Free Unlearning:
- Weight-Space Linear Unlearning (WLU): Uses closed-form linear operations (e.g., task-vector subtraction, weight negation) on a reference checkpoint to suppress unwanted behavior without retraining.
- Representation Projection Unlearning (RPU): Projects internal activations or attention outputs away from a learned subspace associated with the forget signal, effectively filtering out target concepts without modifying model weights.
Decoding Time Unlearning:
- Guidance-Path Control: Modifies the sampler or score function during generation (e.g., Dynamic Negative Guidance) to steer outputs away from unsafe concepts while keeping the base model fixed.
- Conditioning-Path Control: Transforms the conditioning signal (e.g., text embeddings) before it enters the model, effectively weakening the condition for the target concept without altering model parameters.

3. Key Contributions

The paper makes three primary contributions to the field of multimodal unlearning:

Foundational Survey: It synthesizes the state of multimodal unlearning across image, text, video, and audio foundation models, providing a unified framework that covers mechanisms, theory, and evaluation.
System-Level Lens: It introduces a system-first taxonomy organized by intervention stage and control pathway. This contrasts with prior algorithm-centric surveys, enabling clearer comparisons across different model classes (VLMs, DMs, AFMs) and clarifying trade-offs between deletion strength, utility retention, efficiency, and reversibility.
Emerging Frontiers: It outlines critical open challenges, including the need for robust evaluation metrics, adversarial robustness against reactivation, and practical deployment constraints.

4. Results and Evidence

As a survey paper, the primary "results" are the systematic comparisons and analyses of existing literature rather than new experimental data. The paper presents:

A Curated Repository: A comprehensive collection of methods, datasets, and benchmarks.
Dataset and Benchmark Analysis: The paper reviews key datasets used for identity, affect, video, and copyright unlearning (e.g., CelebA, VoxCeleb, DreamBooth) and evaluates existing benchmarks (e.g., MU-Bench, MLLMU-Bench, PEBench). It highlights that current benchmarks often lack unified standards, relying on synthetic identities or narrow model families.
Evaluation Framework: It consolidates metrics for forget quality (e.g., Unlearning Accuracy, CLIP-Sim Drop), safety (Refusal Rate), privacy (Membership Inference Attacks), and utility (Top-k Accuracy, FID, LPIPS).
Trade-off Analysis: The survey clarifies that methods vary significantly in their trade-offs; for instance, training-free methods offer efficiency but may lack the deletion strength of retraining-based approaches, while architecture editing preserves utility better but requires precise localization of the forget signal.

5. Significance and Future Directions

The paper argues that multimodal unlearning is a central tool for model governance, essential for deploying accountable MFMs in real-world settings. Its significance lies in:

Enabling Selective Removal: Providing mechanisms to delete specific identities, attributes, or concepts without the prohibitive cost of full retraining.
Supporting Compliance: Addressing regulatory requirements like the GDPR (Right to be Forgotten) and copyright laws.
Enhancing Safety: Mitigating the generation of harmful, biased, or unsafe content across modalities.

The paper identifies several open challenges and future directions:

Temporal and Dynamic Modalities: Extending unlearning to streaming audio, video, and motion generation remains an open problem.
Frontier-Scale Models: Scaling unlearning methods to foundation-scale models with billions of parameters is currently limited by narrow evaluations on smaller backbones.
Continual Unlearning: Developing protocols that remain effective under repeated deletions and downstream fine-tuning, preventing the "resurgence" of forgotten concepts.
Cross-Modal Leakage: Addressing the failure of safety or privacy alignment in one modality (e.g., text) to generalize to others (e.g., vision or audio).
Unified Benchmarks: The field lacks standardized, unified benchmarks that span diverse modalities and architectures, making it difficult to compare methods fairly.

In conclusion, this survey provides a structured roadmap for researchers and practitioners to navigate the complexities of multimodal unlearning, emphasizing the need for robust, scalable, and theoretically grounded approaches to ensure the safe and compliant deployment of foundation models.