🤖 AI

MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

This paper introduces MLUBench, a large-scale benchmark for evaluating lifelong unlearning in multimodal large language models, and proposes LUMoE, a method that effectively mitigates the cumulative degradation and multimodal alignment challenges inherent in sequentially removing specific content.

Original authors: He Li, Haoang Chi, Qizhou Wang, Yunxin Mao, Zhiheng Zhang, Jie Tan, Tongliang Liu, Wenjing Yang, Bo Han

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: He Li, Haoang Chi, Qizhou Wang, Yunxin Mao, Zhiheng Zhang, Jie Tan, Tongliang Liu, Wenjing Yang, Bo Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Forgetful" Robot

Imagine you have a super-smart robot assistant (an MLLM) that has read almost everything on the internet and seen billions of pictures. It knows everything: who directed Star Wars, what a giraffe looks like, and the history of the Eiffel Tower.

Now, imagine a user says, "Hey, I want you to forget everything about Star Wars." The robot tries to delete that knowledge. Later, another user says, "Please forget everything about The Matrix." Then, "Forget about Elon Musk." Then, "Forget about the Eiffel Tower."

This is the problem the paper tackles: Lifelong Unlearning. It's not just about forgetting one thing once; it's about a robot having to forget a long list of things, one after another, over time, without losing its ability to answer other questions or becoming a gibberish-speaking mess.

The Problem: The "Broken Glass" Effect

The researchers found that current methods for making robots forget are like trying to fix a broken vase by hitting it with a hammer.

The Current Approach: To make the robot forget "Star Wars," existing methods tweak the robot's internal brain (its weights) to erase that specific memory.
The Side Effect: Every time you tweak the brain to erase one memory, you accidentally damage other parts of the brain.
The Result: After the robot forgets the first few things, it starts to fail at everything else. It might forget how to speak properly, stop recognizing faces, or start hallucinating (making things up). The paper calls this "cumulative degradation." It's like peeling layers off an onion; eventually, there's nothing left, and the robot stops working.

The Unique Challenge: The "Tightrope"

The paper highlights a special problem with Multimodal Large Language Models (MLLMs)—robots that see and speak.

Think of an MLLM as a tightrope walker holding a pole. One end of the pole is "Vision" (seeing images), and the other is "Language" (speaking words). They have to stay perfectly balanced (aligned) to work.

If you try to erase a memory by only tweaking the "Vision" side, the "Language" side gets thrown off balance.
If you tweak the "Language" side, the "Vision" side falls.

The paper claims that you cannot just fix one side without breaking the whole tightrope. This is different from regular text-only robots, where you might just edit the text memory without worrying about images.

The Solution: The "Swiss Army Knife" (LUMoE)

To solve this, the authors built a new method called LUMoE (Lifelong Unlearning with a Mixture-of-Experts).

Instead of trying to surgically remove memories from the robot's main brain (which causes damage), LUMoE uses a different strategy:

The Main Brain stays frozen: The robot's core knowledge remains untouched and safe.
Special "Helpers" (Experts): For every specific thing the robot needs to forget (like Star Wars), the system creates a tiny, specialized "helper" module (a LoRA adapter).
The Gatekeeper: When someone asks a question, a smart "Gatekeeper" checks: "Is this about Star Wars?"
- Yes? The Gatekeeper hands the question to the Star Wars helper, which says, "I don't know about that," and blocks the answer.
- No? The Gatekeeper lets the question go straight to the main brain, which answers normally.

The Analogy: Imagine a library. Instead of burning books to make them disappear (which ruins the library), you just put a "Do Not Enter" sign on the specific shelf for Star Wars. The rest of the library remains perfect, and you can add new "Do Not Enter" signs for new requests without damaging the building.

The New Test: MLUBench

Before this paper, there was no good way to test if robots could handle this "long list of forgetting" problem. Existing tests were too small (like only testing 20 things) or too narrow (only testing faces).

The authors created MLUBench, a massive new test suite:

127 Real-World Entities: From famous people (Elon Musk) to movies (Inception), animals, and buildings.
9 Categories: It covers a wide variety of topics.
5,000+ Images and 15,000+ Questions: A huge dataset to stress-test the robots.

They used this test to prove that old methods fail miserably (scores drop to near zero) while their new LUMoE method keeps the robot smart and functional, even after forgetting dozens of things.

Summary of Findings

Old methods fail: Trying to make a robot forget things one by one using current methods destroys its ability to think and see.
Balance is key: In robots that see and speak, you must protect the connection between sight and speech, or the whole system crashes.
LUMoE works: By using a "Gatekeeper" to route questions to specialized "helpers" instead of rewriting the main brain, the robot can forget specific things while staying smart about everything else.

The paper concludes that this "modular" approach (using helpers instead of brain surgery) is the key to making AI that can respect privacy requests without losing its mind.

Technical Summary: MLUBench and LUMoE for MLLM Lifelong Unlearning

1. Problem Formulation

The paper addresses Multimodal Large Language Model (MLLM) Lifelong Unlearning, a practical yet underexplored challenge where MLLMs must sequentially forget specific multimodal information (e.g., specific entities, images, or concepts) upon user request while retaining general capabilities and knowledge of non-targeted data.

Unlike standard machine unlearning (a one-time removal) or unimodal LLM unlearning, MLLM lifelong unlearning presents a unique and critical constraint: the preservation of multimodal alignment. The authors argue that MLLMs rely on a delicate alignment between vision encoders, language models, and multimodal projectors. Sequential unlearning operations that modify weights in one modality can catastrophically disrupt this alignment, leading to a collapse in overall model performance, even if the unlearning target is unimodal.

The objective is defined as minimizing the cumulative performance degradation on previously unlearned tasks while effectively unlearning new tasks in a sequence $T = \{t_1, t_2, ..., t_k\}$ .

2. Methodology: LUMoE

To address the challenges of cumulative degradation and alignment disruption, the authors propose LUMoE (Lifelong Unlearning with a Mixture-of-Experts). Unlike traditional methods that iteratively update the base model's weights, LUMoE isolates unlearning modifications from the stable base model.

Core Components:

Task-Specific LoRA Adapters: Instead of updating the base MLLM, LUMoE trains lightweight Low-Rank Adaptation (LoRA) adapters for each specific unlearning task. These adapters are trained using Preference Optimization (PO) to learn a "refusal" behavior (e.g., "I cannot answer this") for the forgetting set, rather than attempting to erase knowledge via gradient ascent which may corrupt other weights.
Gate Module (Router): A powerful router model (initially GLM-4V-Plus) dynamically directs inputs. It performs:
- Entity Extraction: Identifies the entity in the input (image/text).
- Task Matching: Checks if the entity belongs to a previously unlearned task's forgetting set.
- Routing: If a match is found, the corresponding LoRA adapter is merged into the base model for inference. If no match is found (i.e., the input belongs to the retained set), the original base model processes the input directly, preserving utility.
Error Handling: If the router is uncertain about an entity, the input is treated as a retained question and processed by the base model to avoid accidental forgetting.

3. Key Contributions

A. MLUBench: A Comprehensive Benchmark

The authors introduce MLUBench, a large-scale benchmark designed specifically to evaluate MLLM lifelong unlearning.

Scale & Diversity: It covers 127 real-world entities across 9 distinct classes (Animals, Astronomy, Buildings, Cartoons, Corporations, Movies, Personage, Plants, TV Series).
Data Volume: The dataset includes 5,105 images and 15,414 VQA pairs.
Sequential Structure: Entities are partitioned into a sequence of four unlearning tasks (A, B, C, D) to simulate the lifelong setting.
Generality: Includes semantically diverse question variants to test robustness against prompt variations.

B. Empirical Findings

Using MLUBench, the authors conducted extensive experiments revealing two critical insights:

Severe Cumulative Degradation: Existing unlearning methods (Gradient Ascent, Gradient Difference, KL Minimization, Negative Preference Optimization) suffer from severe, cumulative performance degradation. As the number of sequential unlearning tasks increases, both forget quality (ability to suppress target knowledge) and model utility (ability to answer retained questions) collapse toward zero.
The Multimodal Alignment Challenge: The paper empirically demonstrates that unlearning in MLLMs is fundamentally different from unimodal LLMs. Experiments isolating updates to either the LLM backbone or the vision adapter showed that modifying either component independently leads to catastrophic performance drops. This confirms that preserving multimodal alignment is the unique, fundamental challenge of MLLM lifelong unlearning.

C. Proposed Solution (LUMoE)

LUMoE is presented as a simple yet effective baseline that mitigates these issues. By keeping the base model frozen and using switchable adapters, it prevents the cumulative corruption of weights and the breakdown of multimodal alignment.

4. Experimental Results

Performance: On MLUBench, LUMoE maintains high Forget Quality (near 1.0) and Model Utility (near 0.9) across all sequential tasks. In contrast, baseline methods (GA, GD, KL, NPO) see their scores drop to near-zero after just a few unlearning steps.
General Capabilities: LUMoE preserves the model's general reasoning abilities on external benchmarks (TruthfulQA, MMBench), with performance drops consistently below 0.6%. Baselines cause catastrophic failure on these general tasks.
Robustness: LUMoE demonstrates robustness against jailbreak attacks and maintains performance across different task orderings and varying numbers of tasks (tested up to 5 tasks).
Efficiency: The computational cost is manageable, with adapter training taking ~11 minutes per task and task matching taking ~2 seconds.

5. Significance and Claims

The paper positions its work as a foundational step in understanding and solving MLLM lifelong unlearning.

Benchmarking: It fills a critical gap by providing the first large-scale, diverse benchmark (MLUBench) capable of evaluating the cumulative effects of sequential unlearning in multimodal settings.
Theoretical Insight: It establishes that MLLM lifelong unlearning is not merely an extension of LLM unlearning but a distinct problem constrained by multimodal alignment, a finding that challenges the applicability of existing unimodal unlearning methods.
Methodological Direction: While modest about LUMoE being a "perfect" solution, the authors claim it validates the isolation principle (separating unlearning modifications from the base model) as a viable and effective strategy for preserving model stability and alignment in lifelong scenarios.

The work aims to facilitate future research by providing a standardized evaluation framework and a strong baseline method that addresses the specific complexities of multimodal unlearning.