Technical Summary: On the Robustness of Machine Unlearning for Vision-Language Models

Problem Statement

Vision-Language Models (VLMs) are increasingly deployed in real-world applications but risk memorizing undesirable information from their training data, including sensitive, private, copyrighted, or otherwise harmful content. While machine unlearning has been extensively studied for Large Language Models (LLMs), its application to VLMs presents unique challenges. Unlike text-only models, VLMs store information through entangled visual and textual representations across multiple components (vision encoders, text encoders, and multimodal decoders).

The core problem addressed is the robustness of current VLM unlearning methods. Existing evaluations often rely on standard prompting, which may only verify that a model suppresses direct queries for forgotten entities. However, it remains unclear whether unlearning truly erases multimodal knowledge or merely hides it, leaving residual associations that can be reactivated through alternative queries, contextual cues, or downstream retraining.

Methodology

1. Taxonomy and Review

The authors first establish a comprehensive taxonomy of existing VLM unlearning methods, categorizing them based on the components optimized or modified during the unlearning process:

• Full-Parameter Finetuning: Updates all model parameters ($\theta_v, \theta_t, \theta_d$). Includes adaptations of LLM techniques like Gradient Ascent (GA), Gradient Descent (GD), and Negative Preference Optimization (NPO), as well as VLM-specific extensions that account for visual-textual alignment.
• Vision-Encoder Finetuning: Restricts updates to the vision encoder ($\theta_v$) while keeping text and fusion modules fixed. This targets the source of undesirable visual associations directly.
• Selective Parameter Finetuning: Updates only a subset of parameters identified as critical for the target knowledge, aiming to balance efficiency and effectiveness.
• Inference-Time Intervention: Modifies inputs or hidden activations during inference without updating parameters (e.g., feature masking, activation steering). Note: This category was excluded from the robustness experiments due to the requirement of retraining for the proposed attacks.

2. Robustness Evaluation Framework

To determine if knowledge is truly erased or merely suppressed, the paper proposes three attack paradigms designed to reactivate forgotten multimodal knowledge:

• In-Context Attack: During inference, semantically related but non-explicit contextual cues (e.g., describing a person's profession without naming them) are prepended to the query. This tests whether latent multimodal associations can be reactivated without parameter updates.
• In-Distribution (InD) Attack: The unlearned model is fine-tuned using a small proportion of data sampled from the original forget distribution. The supervision labels are generated by the original pre-trained model. This simulates an adversary with access to the original data attempting to restore the knowledge.
• Out-of-Distribution (OOD) Attack: The unlearned model is retrained on a semantically different but related dataset (e.g., general images from PACS instead of the specific face identities). This tests whether residual representations can be reactivated through distributional transfer.

3. Experimental Setup

• Model: Qwen2.5-VL-3B-Instruct.
• Dataset: VGGFace2, partitioned into a forget set (3 celebrity identities) and a retain set (7 identities).
• Baselines: A wide range of methods from the three active categories (Full-Parameter, Vision-Encoder, and Selective Finetuning).
• Metrics: Forgetting score ($1 - \text{Accuracy}_{\text{forget}}$), Retention score ($\text{Accuracy}_{\text{retain}}$), and General Utility (performance on MMStar, OCRBench, MMMU, RealWorldQA). Evaluations were conducted under Original, Paraphrased, and Discriminative prompt settings.

Key Results

Initial Performance vs. Robustness

• Trade-offs: Full-parameter methods generally achieved high forgetting scores on original prompts but suffered significant degradation in retention and general utility, often exhibiting catastrophic forgetting.
• Discriminative Prompts: Many methods that appeared successful under generative queries failed to suppress discriminative recognition (e.g., "Is this person X?"), suggesting underlying visual associations remained intact.
• Vision-Encoder Advantage: Methods focusing on the vision encoder (e.g., HFRU, HFRU-SFT) demonstrated more balanced performance, maintaining high forgetting and retention scores across prompt types.

Vulnerability to Attacks

• In-Context Attacks: Contextual cues successfully reactivated forgotten knowledge across most paradigms. For instance, methods like RMU, SatImp, and WGA showed noticeable degradation in forgetting performance when contextual descriptions were provided, indicating they primarily suppressed direct retrieval rather than removing associations.
• In-Distribution Attacks: Retraining with as little as 1% of the original forget data caused a rapid recovery of forgotten knowledge for many methods (e.g., SLUG's unlearning effectiveness dropped by over 80%). This suggests that many unlearning processes perform shallow suppression rather than fundamental representation removal. Mmunlearner showed the highest robustness, requiring 50-75% retraining data to significantly degrade performance.
• Out-of-Distribution Attacks: Contrary to the InD attack, OOD retraining did not restore forgotten knowledge; instead, it consistently harmed retention performance across all methods. This indicates that while residual representations exist, they are not easily reactivated by unrelated data distributions.

Key Contributions

1. Systematic Taxonomy: The paper provides the first comprehensive categorization and review of VLM unlearning methods based on their optimization strategies.
2. Unified Evaluation: It conducts a thorough assessment of these methods under multiple prompt settings (Original, Paraphrased, Discriminative), revealing inconsistencies in standard evaluations.
3. Robustness Analysis: The study introduces the first systematic investigation into the robustness of VLM unlearning via three attack paradigms (In-Context, InD, OOD).
4. Empirical Findings: Extensive experiments demonstrate that current approaches often hide rather than fully remove target knowledge, highlighting the need for more reliable multimodal unlearning strategies.

Significance and Claims

The paper claims that the current state of VLM unlearning is insufficient for ensuring the permanent removal of undesirable information. The authors argue that evaluating unlearning solely under standard prompting conditions overestimates the extent of knowledge removal. By demonstrating that forgotten knowledge can be easily reactivated through contextual prompting or minimal retraining on the original data distribution, the study underscores a critical gap between "suppression" and "erasure."

The work concludes that while some methods (particularly those targeting the vision encoder) show promise, the field lacks robust strategies that guarantee the elimination of multimodal associations. The authors emphasize the necessity for future research to develop unlearning techniques that are resilient against these reactivation attacks, ensuring that sensitive or undesirable information is truly removed from the model's latent space.

Note: The paper explicitly states limitations, including the use of a unified evaluation protocol that may simplify implementation details for consistency, and the manual construction of contextual prompts for the In-Context attack, suggesting potential for more automated strategies in future work.