💬 NLP

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

This paper proposes Stochastic Meta-Unlearning (SMU), a bilevel framework that leverages VLM-level feedback to optimize language backbone unlearning, effectively resolving the issue where text-only feedback fails to prevent target recovery in vision-language models while achieving superior forget-retain trade-offs and transferability.

Original authors: Zijie Liu, Jinhao Duan, Gaowen Liu, Sijia Liu, Tianlong Chen

Published 2026-07-22

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zijie Liu, Jinhao Duan, Gaowen Liu, Sijia Liu, Tianlong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can read books and look at pictures. This robot is made of two main parts: a "brain" that understands language (like a very advanced text reader) and a "camera" that sees the world. Together, they are called a Vision-Language Model (VLM). They are amazing at helping us, but sometimes, we need them to forget specific things—maybe a private secret, a harmful idea, or a mistake they learned. This process is called "machine unlearning."

Think of unlearning like trying to erase a specific memory from your friend's brain without deleting their entire personality or making them forget how to speak or see. If you just tell the robot, "Forget that story," it might work fine when you talk to it. But here's the tricky part: if you show the robot a picture related to that story, it might suddenly remember the story again! This happens because the robot's "brain" and "camera" are so tightly connected that the picture can trigger the memory the text tried to erase. Scientists are trying to figure out how to make sure the robot forgets the bad stuff no matter how you ask it—whether you use words, pictures, or both.

This paper tackles that exact puzzle. The researchers noticed a surprising glitch: if you try to make the robot forget something using only text, it often fails when you show it a picture later. The text-only "forgetting" wasn't strong enough to stop the picture from bringing the memory back. To fix this, they invented a clever new training method called Stochastic Meta-Unlearning (SMU).

Instead of just teaching the robot to forget based on text, SMU acts like a strict coach who watches the whole team. Here's how it works: The coach first tells the "language brain" to forget a specific thing using only text (the inner loop). But then, instead of just checking if the text is gone, the coach immediately shows the robot a picture and asks, "Do you still remember?" (the outer loop). If the robot remembers the bad thing because of the picture, the coach goes back and tweaks the starting point of the training so that the next time the brain tries to forget, it does a better job of blocking the memory even when pictures are involved.

The paper suggests that this "two-step" approach is much better than previous methods. When they tested it on real robot brains (using models like Qwen and InternVL) and datasets of memes (like "Hateful Memes" and "Covid Memes"), the results were impressive. Compared to the best existing methods, SMU reduced the robot's ability to recall the forgotten target by an average of 10.52 points (lower is better for forgetting). At the same time, it kept the robot's general knowledge much stronger, improving its ability to remember other things by 20.10 points and its general test scores by 17.01 points.

Even cooler, the researchers found that this method is flexible. It didn't just work for the specific bad memories they trained it on; it could handle new bad memories it had never seen before. It also worked even if they changed the specific rules used to teach the robot to forget. This suggests that by training the robot's brain with the "whole team" (text and image) in mind, they created a more reliable and adaptable way to make AI forget what it shouldn't know, without breaking everything else it knows.

Technical Summary: Stochastic Meta-Unlearning (SMU)

1. Problem Statement

Machine unlearning for Vision-Language Models (VLMs) presents a unique challenge distinct from unlearning in standalone Large Language Models (LLMs). VLMs integrate a language backbone with visual encoders and projection modules into a tightly coupled system. The paper identifies a critical "component-system mismatch" in existing approaches:

The Phenomenon: A target concept successfully "forgotten" by the language backbone when queried via text-only prompts can often be recovered when the same backbone is recomposed with frozen visual modules and queried with image-conditioned prompts.
The Limitation of Current Methods:
- Direct VLM Unlearning: Optimizing the entire VLM directly ensures multimodal forgetting but risks degrading unrelated capabilities (utility) because updates propagate unpredictably across modalities.
- Language-Backbone Unlearning: Updating only the language backbone is computationally efficient and preserves visual modules, but standard unlearning objectives (optimized on text-only data) fail to guarantee that forgetting persists after the backbone is re-integrated with the visual system.

The core challenge is to preserve the utility of language-backbone unlearning while ensuring that the forgetting behavior holds true in the fully composed, multimodal VLM.

2. Methodology: Stochastic Meta-Unlearning (SMU)

The authors propose Stochastic Meta-Unlearning (SMU), a bilevel meta-optimization framework designed to align language-backbone updates with VLM-level forgetting behavior.

2.1 Bilevel Framework

SMU treats VLM unlearning as a meta-learning problem with two distinct loops:

Inner Loop (Text-Side Unlearning):
- Operates solely on the language backbone parameters ( $\theta_T$ ).
- Applies $K$ steps of unlearning using only unimodal text data (forget and retain sets).
- The visual encoder and projection modules ( $\phi$ ) remain frozen and are not updated.
- This step produces an updated backbone $\theta_{T,K}$ .
Outer Loop (VLM-Level Feedback):
- Recomposes the updated backbone $\theta_{T,K}$ with the frozen visual components $\phi$ to form the full VLM $F_{\theta_{T,K}, \phi}$ .
- Evaluates the model on multimodal queries (image + text) and text-only queries.
- Computes a loss ( $L_{outer}$ ) based on forgetting performance (target removal) and utility preservation (retain/test performance) at the VLM level.
- Backpropagates the gradient of this outer loss through the inner unlearning steps to update the pre-unlearning initialization ( $\theta_T$ ).

2.2 Stochastic Task Sampling

To address generalization, SMU does not optimize for a single fixed forgetting target or unlearning operator. Instead, during meta-training, it stochastically samples environments $e = (z, m, c)$ from a distribution $p(E)$ , where:

$z$ : The forgetting target.
$m$ : The inner-loop unlearning operator (e.g., GradDiff, NPO).
$c$ : The query condition.

By minimizing the expected outer loss over these sampled environments, SMU learns an initialization that is robust to changes in the target, the unlearning algorithm, and the query modality.

2.3 Meta-Test Phase

At deployment (meta-test time), given a new deletion request:

The system starts from the learned initialization $\theta^*_T$ .
It performs standard text-side unlearning steps using the new target data.
The resulting backbone is plugged into the frozen VLM.
The final model is evaluated on both text-only and multimodal test sets.

3. Key Contributions

Identification of Component-System Mismatch: The paper empirically demonstrates that text-only unlearning is insufficient for VLMs, as targets forgotten in the language backbone can be recovered via visual inputs.
Stochastic Meta-Unlearning (SMU): A novel bilevel framework that uses VLM-level feedback to meta-learn an initialization for the language backbone. This ensures that local backbone updates result in global multimodal forgetting.
Generalization via Stochasticity: The introduction of stochastic task sampling allows the learned initialization to transfer to unseen forgetting targets and different unlearning operators, avoiding overfitting to specific deletion scenarios.

4. Experimental Results

The authors evaluated SMU on two VLMs (Qwen2.5-VL and InternVL3) and two multimodal meme datasets (Hateful Memes and COVID Memes), comparing against three baselines: GradDiff, NPO, and RMU.

Forget-Retain Trade-off: SMU achieved the best overall balance. Compared to the strongest baseline for each metric, SMU reduced average Forget accuracy by 10.52 points while improving average Retain accuracy by 20.10 points and Test accuracy by 17.01 points.
Multimodal Robustness: Unlike baselines that either failed to forget under image conditions or destroyed utility, SMU successfully removed targets in the recomposed VLM while maintaining high performance on unrelated tasks.
Transferability:
- Target Transfer: SMU trained on one target successfully generalized to new, unseen forgetting targets.
- Operator Transfer: SMU trained with one unlearning operator (e.g., GradDiff) remained effective when tested with different operators (e.g., RMU, NPO).

5. Significance and Claims

The paper claims that SMU provides a more reliable and transferable solution for VLM unlearning by bridging the gap between component-level updates and system-level behavior.

Reliability: By using VLM-level feedback to guide the initialization, SMU ensures that forgetting is not an artifact of text-only evaluation but holds true in the full multimodal context.
Efficiency: It retains the computational efficiency of updating only the language backbone while avoiding the utility degradation associated with direct VLM optimization.
Practicality: The stochastic design makes the method robust to real-world scenarios where deletion requests vary in content and the specific unlearning algorithm used may differ from training assumptions.

The authors conclude that VLM-level feedback is essential for making language-backbone unlearning effective for multimodal systems, and that meta-learning provides a viable path to achieving this without retraining the entire model or compromising utility.