🤖 machine learning

A Model Merging Approach for Continual MLLM Unlearning

This paper introduces MCU, a model merging approach that dynamically integrates multiple one-shot unlearning adapters into a unified model to effectively address the challenges of continual multimodal unlearning by mitigating interference and preserving utility through cross-task dependency management.

Original authors: Yuhang Wang, Linlin Zhang, Haoxuan Ji, Xianmin Ye, Zhenxing Niu, Haichang Gao

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuhang Wang, Linlin Zhang, Haoxuan Ji, Xianmin Ye, Zhenxing Niu, Haichang Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot friend that can see pictures, read text, and chat about almost anything. This robot is a "Multimodal Large Language Model" (MLLM). It's incredibly talented, but sometimes it learns things it shouldn't—like private secrets, copyrighted stories, or outdated facts. In the world of AI, there's a growing need to teach these robots to "unlearn" specific bad or sensitive information without making them forget everything else they know. Think of it like trying to remove a specific stain from a favorite sweater without shrinking the whole garment or ruining the pattern.

For a long time, scientists tried to do this by hitting the "reset" button on the robot's brain every time a new piece of information needed to be erased. But here's the catch: if you keep hitting reset over and over again, the robot starts to get confused. It might forget the new thing you just asked it to forget, or worse, it might start forgetting things it was supposed to keep, like how to tell a joke or identify a cat. This is called "unlearning rebound" and "retention drift." The robot gets worse at being a robot the more you try to fix it. This paper tackles the messy problem of how to keep a robot clean and sharp when you have to delete a long, never-ending list of secrets one by one.

Enter MCU (Merging for Continual Unlearning), a clever new approach proposed by researchers Yuhang Wang and his team. Instead of constantly rewriting the robot's brain, which causes all that confusion and damage, MCU treats each "forgetting request" like a separate, tiny instruction manual. Imagine you have a stack of sticky notes, each telling the robot to forget a different secret. Old methods would try to paste these notes one by one onto the robot's brain, eventually covering it in a messy, conflicting mess of paper.

MCU takes a different, more organized approach. It gathers all these sticky notes (which the researchers call "adapters") and blends them together into a single, perfect instruction sheet before applying it to the robot. The magic happens in how they mix these notes. The researchers discovered that these "forgetting notes" aren't isolated; they actually talk to each other. Sometimes, two notes help each other forget better (synergy), but other times, they fight and cancel each other out (interference), or they accidentally erase good memories along with the bad ones.

To solve this, MCU acts like a master chef mixing ingredients. First, it looks at the notes and keeps only the most important parts, tossing out the weak, blurry scribbles (this is called "dominant direction selection"). Then, it checks if any single note is shouting too loudly and drowning out the others, and gently turns its volume down (this is "channel capacity control"). Finally, and most importantly, it rearranges the notes so that the helpful ones work together while the fighting ones are separated, ensuring they don't ruin the robot's other skills.

The team tested this on two major benchmarks, ICU-Bench and MLLMU-Bench, which are like giant obstacle courses for AI unlearning. They ran experiments where the robot had to forget up to 100 different things in a row. The results were impressive. While other methods started to crumble, forgetting good things or failing to forget bad things as the list grew longer, MCU stayed steady. For example, after 100 tasks on the Qwen2-VL-7B model, MCU managed to lower the "Forget" score (meaning it successfully erased the bad info) to around 28.7, while keeping the "Retain" score (how well it kept its general smarts) high at 75.6. In contrast, other methods saw their retention scores drop significantly or their unlearning become ineffective.

The paper suggests that by merging these unlearning instructions intelligently rather than applying them sequentially, we can stop the robot from getting "brain fog" after too many deletions. It's a way to keep the AI's memory clean and its mind sharp, proving that you don't have to break the machine to fix its memory. The authors show that this method works without needing to retrain the whole model from scratch, making it a practical tool for keeping AI safe and up-to-date in a world where privacy and copyright matter.

Technical Summary: A Model Merging Approach for Continual MLLM Unlearning

Problem Statement

Multimodal Large Language Models (MLLMs) often encode private, proprietary, or outdated information that necessitates removal via machine unlearning. While existing methods effectively handle one-shot unlearning requests, they struggle in continual scenarios where deletion requests arrive sequentially.

Current sequential approaches typically apply a new one-shot model update or adapter to the current state of the model for each incoming request. As illustrated in the paper, this paradigm couples new requests to preceding model states, leading to three critical issues:

Unlearning Rebound: The model relearns previously forgotten information.
Retention Drift: The model's ability to retain unrelated, useful knowledge degrades over time.
Utility Degradation: General multimodal capabilities suffer due to cumulative parameter drift.

The core challenge is to accommodate an expanding sequence of deletion requests while preserving both historical unlearning effects and general multimodal utility.

Methodology: Merging for Continual Unlearning (MCU)

The authors propose MCU, a framework that reframes continual unlearning as a model-merging problem. Instead of sequentially updating the model, MCU dynamically merges multiple independently trained one-shot unlearning adapters into a unified adapter.

Key Observations

Through a "leave-one-out" merging analysis, the authors discovered cross-task unlearning transfer: adapters trained for specific requests can inadvertently contribute to forgetting targets associated with other requests. This indicates that unlearning adapters exhibit strong cross-task dependencies. These dependencies can be:

Synergistic: Reinforcing unlearning across requests.
Interfering: Weakening target unlearning or damaging retained knowledge.

The MCU Framework

MCU operates in a shared core-space representation to manage these dependencies. The process involves four main stages:

Shared Core-Space Representation:
Instead of manipulating full parameter adapters, MCU projects LoRA adapters (derived from the same base model $M_0$ ) into a shared orthonormal basis. This allows for efficient analysis of spectral structures and cross-task dependencies.
Structure-Preserving Preprocessing:
To reduce complexity and prevent over-concentration, MCU applies two operations:
- Dominant Direction Selection: Retains only the top- $k$ singular components of each adapter, discarding low-energy tail directions that contribute little to the update magnitude.
- Channel Capacity Control: Applies a soft row-capacity constraint to prevent a small number of core-space coordinates from disproportionately dominating the update geometry.
Dependency-Aware Direction Reconfiguration:
This is the core innovation. Rather than enforcing complete orthogonality (which eliminates both interference and beneficial synergy), MCU performs a Gram reconfiguration:
- It constructs Gram matrices of the left and right singular directions.
- It solves an optimization problem to find a new geometry that minimizes deviation from the original structure while enforcing non-negative cross-request similarities for off-diagonal blocks (reducing antagonistic interference) and maintaining identity blocks for within-request directions.
- This step suppresses harmful cross-task interactions while preserving beneficial shared structures that support unlearning transfer.
Direction Recovery and Merging:
The optimized Gram matrices are used to recover direction matrices via an orthogonal Procrustes problem. The processed updates are then merged in the shared core space and mapped back to the original parameter space to form a unified adapter.

Crucially, MCU is a training-free, post-hoc procedure that requires no additional gradient-based optimization once the one-shot adapters are available.

Key Contributions

Framework: Introduced a merging-based framework for continual unlearning that replaces repeated model modifications with the dynamic merging of one-shot adapters.
Discovery: Uncovered and characterized cross-task unlearning transfer and the associated dependencies among unlearning adapters, distinguishing between synergistic and interfering effects.
Algorithm (MCU): Developed a method that preserves dominant task structures, controls over-concentrated coordinates, and reconfigures cross-task dependencies in a shared space to suppress antagonistic interactions while retaining beneficial transfer.
Empirical Validation: Demonstrated through extensive experiments that MCU supports long-horizon continual unlearning with superior effectiveness compared to sequential baselines.

Experimental Results

The authors evaluated MCU on ICU-Bench (100-task sequential protocol) and MLLMU-Bench (5-stage protocol) using Qwen2-VL-7B and LLaVA-1.5-7B.

Unlearning Effectiveness: MCU consistently achieved lower "Forget" scores (indicating better removal of target knowledge) compared to sequential baselines like GA-Diff, KL-Min, and MMUnlearner. For example, on Qwen2-VL-7B at Task 100, MCU reduced Forget VQA accuracy to 28.7 (vs. 33.3 for MMUnlearner) while significantly improving retention.
Retention and Utility: MCU maintained high "Retain" scores and general multimodal utility (measured by Generation Quality and Retain Stability Rate). Unlike some baselines that achieved low Forget scores by causing general model collapse, MCU preserved the model's ability to perform unrelated tasks.
Stability: MCU demonstrated reduced "Forgetting Rebound" (FR) and "Retention Drift" (RSR) over long sequences, confirming its ability to maintain historical unlearning effects without degrading performance over time.
Ablation Studies: Experiments confirmed that all components of MCU (direction selection, capacity control, and dependency-aware reconfiguration) are necessary. Specifically, dependency-aware reconfiguration was shown to be critical for balancing unlearning and retention, outperforming strict orthogonalization methods.

Significance

The paper claims that MCU addresses the fundamental limitation of existing continual unlearning methods: the accumulation of parameter drift and the inability to handle cross-task dependencies. By treating unlearning as a model-merging problem and explicitly modeling the geometry of unlearning adapters, MCU enables the use of simple one-shot adapters for complex, long-horizon continual scenarios. The method offers a practical, training-free solution that preserves both the "forgetting" of sensitive data and the "remembering" of general capabilities, making it suitable for real-world deployment where deletion requests arrive continuously.