Technical Summary: Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant progress in vision-language tasks but pose substantial privacy and safety risks by inadvertently memorizing sensitive personal information, copyrighted content, or unsafe knowledge. Once encoded in model parameters, this knowledge can be triggered by visual or textual queries during inference.

Existing unlearning paradigms face a critical trade-off:

• Training-based methods (e.g., Gradient Ascent, Negative Preference Optimization) update model parameters to remove knowledge but often degrade general model utility and capabilities.
• Training-free methods (e.g., In-Context Unlearning) preserve utility by avoiding parameter updates but fail to remove memorized knowledge at the parameter level. Consequently, they remain vulnerable to reverse-engineering attacks and are insufficient in multimodal settings where strong visual inputs can override textual safety instructions.

The core challenge is to achieve effective parameter-level unlearning in MLLMs while preserving general utility and robustness against adversarial prompts, without relying on external teacher models or explicit undesirable response annotations.

2. Methodology: VGID

The authors propose Visual-Noise Guided In-Context Distillation (VGID), a teacher-student framework designed to bridge the gap between training-free intervention and parameter-level unlearning.

2.1 Dual-Modal Intervention

The core insight of VGID is that textual in-context instructions alone are insufficient to suppress privacy leakage induced by salient visual features. To address this, VGID constructs an unlearning-oriented teacher distribution from a frozen base model using a dual-modal intervention:

1. Visual Perturbation: The input image is perturbed (e.g., via Gaussian noise, random noise replacement, or white-image replacement) to weaken target-sensitive visual evidence.
2. Textual In-Context Instruction: An unlearning instruction (e.g., "refuse to identify the person") is prepended to the textual query.

The frozen base model processes this perturbed multimodal input to generate a soft target distribution ($\pi_T$). This distribution serves as a supervisory signal that captures the desired unlearned behavior (suppressing sensitive tokens, encouraging safe responses) without requiring an externally trained teacher.

2.2 Distillation Framework

VGID employs a student model ($\pi_\theta$) trained to match the teacher distribution through knowledge distillation:

• Forget Objective: For samples in the forget set ($D_F$), the student minimizes the Kullback-Leibler (KL) divergence between its output on the original input and the teacher's output on the perturbed input. This guides the student to learn the unlearning behavior at the parameter level.
• Retain Objective: For samples in the retain set ($D_R$), the student minimizes the KL divergence between its output and the frozen base model's output on the original input. This ensures the preservation of general multimodal capabilities.

The overall loss function is a weighted combination:
$$L_{overall} = \alpha L_{forget} + \beta L_{retain}$$

This approach allows VGID to distill the "unlearning behavior" induced by the dual-modal intervention directly into the model parameters, achieving robustness without retraining from scratch or using external teachers.

3. Key Contributions

• Identification of Modality Gap: The paper demonstrates that purely textual in-context unlearning instructions are insufficient for MLLMs because visual inputs act as dominant conditioning signals that can override text-based safety conditioning.
• Dual-Modal Intervention Strategy: The authors show that combining visual perturbation with textual in-context unlearning provides complementary signals, effectively suppressing target private knowledge more than either modality alone.
• VGID Framework: A novel distillation-based framework that dynamically constructs an unlearning-oriented teacher distribution from a frozen base model. It achieves parameter-level unlearning without external teacher models or explicit refusal annotations.
• Empirical Validation: Extensive experiments on MLLMU-Bench across multiple backbones (Qwen2.5-VL, LLaVA-1.5) demonstrate that VGID achieves strong unlearning effectiveness while maintaining competitive model utility and robustness against reverse-engineering attacks.

4. Experimental Results

Experiments were conducted on MLLMU-Bench with forget ratios of 5%, 10%, and 15% across Qwen2.5-VL-3B, LLaVA-1.5-7B, and LLaVA-1.5-13B.

• Unlearning Effectiveness: VGID significantly reduced target private knowledge. In a representative setting (LLaVA-1.5-7B, 5% forget ratio), VGID reduced the forget set ROUGE-L score from 0.571 (Base) to 0.200, a reduction of 0.371.
• Utility Preservation: Unlike gradient-based baselines that often degrade retain-set performance, VGID maintained strong utility. In the same setting, the retain set ROUGE-L dropped by only 0.055 (from 0.496 to 0.441), and cloze/classification accuracy remained competitive.
• Robustness: VGID demonstrated superior robustness against reverse-engineering attacks. While In-Context Unlearning (ICU) performance recovered significantly when reverse prompts were applied (forget set ROUGE-L increased from 0.462 to 0.518), VGID remained stable (forget set ROUGE-L increased only from 0.200 to 0.220).
• Ablation Studies:
  • Teacher Construction: Dual-modal intervention (VGID) outperformed single-modality interventions (ICU-T or Noise-T), confirming the necessity of combining visual and textual cues.
  • Perturbation Strategy: Random noise replacement proved more effective than Gaussian noise or white-image replacement, suggesting that the perturbation pattern is critical for removing visual cues while maintaining a usable input for the teacher.
  • Prompt Style: "Strict Enforcement" prompts provided the most stable balance between forgetting and utility preservation.

5. Significance and Claims

The paper claims that VGID offers a superior trade-off between unlearning completeness and model utility compared to existing training-based and training-free methods. By distilling an intervention-induced distribution, VGID successfully transfers the robustness of inference-time intervention into the model parameters, making the unlearning behavior resistant to reverse-engineering attacks.

The authors position VGID as a practical solution for privacy-preserving MLLM deployment that avoids the computational cost of retraining and the vulnerability of purely inference-time strategies. They acknowledge limitations, noting that the current scope focuses on privacy-oriented unlearning and that the design space for visual perturbations remains underexplored, suggesting future work could extend this paradigm to copyright protection and unsafe knowledge removal.