🤖 AI

Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearning

This paper proposes Null Space Constrained Contrastive Visual Forgetting, a method that fine-tunes the visual module of Multimodal Large Language Models using contrastive learning and null space constraints to effectively forget target visual knowledge while preserving non-target visual and all textual knowledge in both static and continual unlearning scenarios.

Original authors: Yuhang Wang, Zhenxing Niu, Haoxuan Ji, Guangyu He, Linlin Zhang, Haichang Gao

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhang Wang, Zhenxing Niu, Haoxuan Ji, Guangyu He, Linlin Zhang, Haichang Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, super-observant assistant (a Multimodal Large Language Model, or MLLM) who has memorized a massive library of books and photo albums. This assistant is great at answering questions about people, places, and things.

However, some of the photos and stories in that library are private or sensitive. You want the assistant to forget specific details about a certain person (let's call him "Mr. Smith") without forgetting anything about anyone else, and without losing the ability to read or speak normally.

The problem is that in these AI assistants, the "visual" memory (what Mr. Smith looks like) and the "textual" memory (his name, address, and job) are tangled together like a knot of yarn. If you try to pull out the visual thread, you might accidentally unravel the whole sweater, making the assistant forget everything or stop working properly.

This paper proposes a clever new way to untangle that knot. Here is how they did it, explained simply:

1. The Strategy: Freeze the Brain, Train the Eyes

Most previous methods tried to retrain the entire assistant's brain to forget Mr. Smith. This is like trying to erase a specific memory by rewriting the assistant's entire personality—it's expensive and often ruins other memories.

Instead, the authors decided to freeze the assistant's "brain" (the language part) and only retrain its "eyes" (the visual part).

The Analogy: Imagine the assistant is a detective. The "brain" knows how to speak and reason. The "eyes" are the part that looks at photos to identify who is in them.
The Move: They told the brain, "You stay exactly as you are." They only tweaked the eyes so that when they look at a photo of Mr. Smith, they no longer recognize him. But when they look at photos of other people, they still recognize them perfectly.

2. The First Trick: The "Push and Pull" Game (Contrastive Visual Forgetting)

How do you teach the eyes to forget one person without making them forget everyone?

The authors created a game called Contrastive Visual Forgetting.

The "Push": When the assistant sees a photo of Mr. Smith, the system yells, "Don't look like the old version of yourself! Move your mental image of him far away!" It pushes the representation of Mr. Smith into a different corner of the mind.
The "Pull": But if you just push things away randomly, you might accidentally push away memories of other people too. So, the system also says, "But make sure you stay close to the 'average' safe zone where all the other people live."
The Result: The eyes learn to shove Mr. Smith's image into a "forbidden zone" while keeping all other people's images in the "safe zone." This ensures Mr. Smith is forgotten, but the rest of the library stays intact.

3. The Second Trick: The "Safe Tunnel" (Null-Space Constrained Unlearning)

Even with the Push and Pull game, there's a risk that the changes made to forget Mr. Smith might accidentally damage the memory of a neighbor who looks a bit like him.

To fix this, the authors built a Safe Tunnel (called the Null Space).

The Analogy: Imagine the assistant's knowledge is a giant city. The "Safe Tunnel" is a special underground road that only connects to the parts of the city that don't matter for remembering other people.
The Move: They calculated exactly which directions in the assistant's mind are safe to change. They then forced all the "forgetting" updates to travel only through this Safe Tunnel.
The Result: The assistant can make changes to forget Mr. Smith, but because it's stuck in the tunnel, it physically cannot touch or damage the memories of anyone else. It's like renovating a kitchen without accidentally breaking the plumbing in the bathroom.

4. The Bonus: Handling a Moving Target (Continual Unlearning)

Sometimes, you don't just want to forget one person; you might get a list of ten people to forget, one after another, over time. This is called Continual Unlearning.

Usually, forgetting Person A makes it harder to forget Person B later without ruining everything. But because the authors built their "Safe Tunnel" so carefully, their method can handle a long list of people to forget. Each time they add a new person to the "forget" list, the assistant uses the same safe tunnel to remove that person's image without messing up the previous work or the general knowledge.

The Bottom Line

The paper claims that by freezing the language brain and using these two tricks (Push/Pull and the Safe Tunnel) on the visual eyes, they can:

Forget the specific visual look of a target person very effectively.
Remember everyone else perfectly.
Keep the assistant's ability to read, speak, and reason completely intact.

They tested this on several benchmarks (like MLLMU, UMU, and CLEAR) and found that their method was better at balancing "forgetting" and "remembering" than any previous method, even when the requests came one after another.

Technical Summary: Null-Space–Constrained Contrastive Visual Forgetting for MLLM Unlearning

1. Problem Definition

The paper addresses the challenge of machine unlearning within Multimodal Large Language Models (MLLMs). While unlearning in text-only LLMs aims to remove specific textual knowledge, MLLM unlearning presents a more complex trade-off: the goal is to erase target visual knowledge (e.g., the visual appearance of a specific person) while strictly preserving:

Non-target visual knowledge (visual characteristics of other entities).
All textual knowledge (factual information about both target and non-target entities).

The core difficulty lies in the tight entanglement of visual and textual modalities. Standard unlearning methods that update the entire model often degrade textual capabilities or fail to selectively forget visual concepts without collateral damage to retained visual knowledge. Furthermore, existing methods typically rely on output-level supervision (losses applied to final model outputs), which is inefficient when only the visual module is being updated, as the supervision signal is attenuated by the intervening LLM backbone.

2. Methodology

The proposed framework, Null-Space–Constrained Contrastive Visual Forgetting (NC-CVF), operates by freezing the LLM backbone and fine-tuning only the visual module (Vision Encoder and Projector). The approach consists of three main components:

A. Intermediate-Level Supervision via Contrastive Visual Forgetting (CVF)

Instead of applying supervision at the final output, the method injects supervision directly into the Intermediate Visual Representations (IVRs) produced by the visual module.

Reference Model: The original, pre-unlearning model serves as a reference.
Contrastive Mechanism:
- Push: For target entities (forget set), the IVRs of the unlearned model are pushed away from the reference IVRs using a modified InfoNCE-like loss ( $L_{CVF-push}$ ). This explicitly separates target visual concepts from their original representations.
- Pull: To prevent the model from drifting into a region that harms retained knowledge, the unlearned IVRs are pulled toward an anchor representation ( $p^a_r$ ). This anchor is the mean of reference IVRs from retained samples, ensuring the model stays within a "safe" feature space for non-target knowledge.
Retention Loss: An explicit Mean Squared Error ( $L_{RET}$ ) is applied to the IVRs of retained samples to ensure they match the reference model.
General Utility: A standard output-level likelihood loss ( $L_{GUM}$ ) is included on retained data to maintain general multimodal capabilities.

B. Null-Space–Constrained Unlearning (NCU)

To further decouple forgetting from retention, the method constrains the parameter updates to a subspace that is orthogonal to the directions encoding retained knowledge.

Subspace Identification: Using Singular Value Decomposition (SVD) on the activation covariance matrix of the retained set, the method identifies the subspace spanned by the eigenvectors corresponding to the smallest eigenvalues. These directions are weakly associated with retained knowledge.
LoRA Constraint: The unlearning is implemented via Low-Rank Adaptation (LoRA). The method initializes the LoRA matrix $A$ using the basis of this null space and freezes it throughout training. Only the matrix $B$ is optimized.
Effect: This ensures that the weight updates ( $\Delta W = BA$ ) have negligible impact on the layer outputs for retained data ( $A x \approx 0$ ), effectively isolating the forgetting operation from knowledge retention.

C. Continual Unlearning Extension

The framework is extended to continual unlearning, where forgetting requests arrive sequentially. By freezing the null-space basis $A$ and initializing subsequent tasks with the optimized $B$ from the previous task, the method prevents cumulative degradation of retained knowledge over time.

3. Key Contributions

Intermediate-Level Supervision: The paper proposes shifting unlearning supervision from the final output to the intermediate visual representations, addressing the signal attenuation problem in MLLMs when only the visual module is updated.
Contrastive Visual Forgetting (CVF): A novel mechanism that uses a "push-pull" strategy in the feature space to separate target visual concepts while anchoring them to retained knowledge regions.
Null-Space Constrained Unlearning (NCU): A structural constraint that explicitly decouples forgetting and retention by restricting updates to a subspace orthogonal to retained knowledge, inspired by null-space learning in LLM editing.
Continual Unlearning Framework: The first approach to address sequential forgetting requests in MLLMs, demonstrating that the proposed constraints mitigate the cumulative degradation typically seen in continual settings.

4. Experimental Results

The method was evaluated on three benchmarks: MLLMU-Bench, UMU-Bench, and CLEAR, using LLaVA-1.5-7B and Qwen2-VL-7B models.

Forgetting Effectiveness: The method achieved the lowest accuracy on "Forget VQA" tasks (indicating successful removal of target visual knowledge) across all benchmarks, outperforming state-of-the-art baselines like MMU and MANU.
Knowledge Retention: It maintained high accuracy on "Retain VQA" and "Forget/Retain QA" tasks. Notably, it preserved textual knowledge (QA tasks) at levels comparable to the vanilla model, whereas other methods often suffered significant drops.
Continual Unlearning: In sequential forgetting scenarios, the proposed method exhibited significantly slower degradation in retained and real-world utility compared to baselines, which showed severe performance collapse after a few tasks.
Efficiency: By freezing the LLM and using a pre-computed null-space constraint, the method reduced training time and GPU memory usage compared to full-model fine-tuning baselines.

5. Significance and Claims

The paper claims to present a robust solution to the forgetting-retention trade-off in MLLMs. Its significance lies in:

Selective Unlearning: Successfully demonstrating that visual knowledge can be erased without compromising the model's ability to recall textual facts or recognize other visual entities.
Structural Decoupling: Moving beyond simple loss-function adjustments to structural constraints (null-space) that mathematically isolate the forgetting process from retained knowledge.
Continual Viability: Addressing a practical, under-explored scenario where multiple unlearning requests occur over time, showing that the method scales effectively without cumulative damage.

The authors position this work as a step toward making MLLMs safer and more compliant with privacy regulations by enabling precise, modality-specific knowledge removal without the prohibitive cost of retraining.