Rethinking Post-Unlearning Behavior of Large Vision-Language Models

1. Problem Statement

Large Vision-Language Models (LVLMs) trained on web-scraped datasets often memorize sensitive personal information about individuals (e.g., names, professions, biographies). While Machine Unlearning aims to remove this specific knowledge (the "forget target") from the model, existing methods focus almost exclusively on suppression (preventing the model from outputting the private data).

The authors identify a critical gap: existing methods rarely prescribe what the model should output instead of the forgotten content. This leads to "Unlearning Aftermaths," characterized by three undesirable behaviors:

1. Degeneration: The model outputs repetitive tokens or empty strings.
2. Hallucination: The model generates confident but factually incorrect or fabricated information about the entity.
3. Excessive Refusal: The model simply refuses to answer any query related to the entity, degrading user experience.

The core problem is that current unlearning techniques sacrifice model utility and safety (by hallucinating) to achieve privacy, failing to provide a high-quality, informative, and privacy-preserving alternative response.

2. Methodology: PUBG (Post-Unlearning Behavior Guidance)

The authors propose PUBG, a novel unlearning framework that explicitly guides the model's behavior toward a desirable output distribution after unlearning, rather than relying on naive suppression.

A. Task Formulation

The goal is defined as an Entity Unlearning Task:

• Input: An image of a "forget-entity" and an open-ended query (e.g., "Tell me about this person").
• Constraint: The model must not generate private details (name, profession, bio).
• Objective: The model must generate an informative response based only on visually observable features (e.g., clothing, hair, posture).

B. Reference Distribution Construction

To guide the model, PUBG constructs a Reference Distribution ($p_{\theta^*}$) representing the desired behavior.

• Mechanism: The authors leverage the strong instruction-following and in-context editing capabilities of the original (pre-unlearning) LVLM.
• Process: They use an in-context prompt (see Figure 2 in the paper) instructing the original model to "forget" the entity's identity and describe only visual attributes.
• Result: This generates high-quality, privacy-safe, visually grounded descriptions that serve as the target distribution for the unlearning process.

C. Optimization Objective

PUBG optimizes the model parameters ($\theta$) using a combined loss function consisting of two components:

1. Gradient Ascent Loss ($L_{GA}$):

   • Standard unlearning loss that increases the probability of the private information (the "forget" set) to suppress it.
   • $L_{GA}(\theta) = \mathbb{E}[\log p_\theta(rei,k | I_{ei}, q)]$

2. Behavior Guidance Loss ($L_{BG}$):

   • A new loss function that minimizes the Kullback-Leibler (KL) divergence between the unlearned model's output distribution and the reference distribution ($p_{\theta^*}$).
   • Practically, this is implemented by minimizing the cross-entropy between the reference output ($o^*$) and the unlearned model's prediction.
   • $L_{BG}(\theta) = \mathbb{E}[-\log p_\theta(o^* | I_{ei}, q)]$

Total Objective:
$$\arg \min_\theta (L_{GA}(\theta) + L_{BG}(\theta))$$

This dual-objective approach ensures the model forgets the private data (via $L_{GA}$) while learning to generate the specific type of visual description defined by the reference distribution (via $L_{BG}$).

3. Key Contributions

1. Identification of "Unlearning Aftermaths": The paper formally defines and quantifies the degradation in model behavior (degeneration, hallucination, refusal) that occurs when existing unlearning methods are applied to generative LVLMs.
2. New Task Definition: They introduce a task formulation that requires models to provide visually grounded, informative alternatives rather than just suppressing outputs.
3. PUBG Framework: The proposal of a novel method that uses an in-context prompted reference distribution to guide post-unlearning behavior, effectively balancing privacy and utility.
4. Comprehensive Evaluation: Extensive experiments demonstrating that PUBG outperforms state-of-the-art baselines in maintaining informativeness and reducing hallucinations while preserving privacy.

4. Experimental Results

The authors evaluated PUBG on LLaVA-1.6-Mistral and LLaVA-1.6-Vicuna (7B parameters) using the Celeb-1000 dataset. They compared PUBG against four baselines: Gradient Ascent (GA), Negative Preference Optimization (NPO), Random Tuning (RANDOM), and Rejection Tuning (REJECT).

Key Findings:

• Privacy Preservation: All methods, including PUBG, achieved a perfect Unlearning Success Rate (USR = 1.0), meaning no private information was leaked.
• Mitigation of Aftermaths:
  • Baselines: GA and NPO produced degenerate/empty outputs; RANDOM caused severe hallucinations (JUDGE_hall score ~5.0); REJECT resulted in generic refusals.
  • PUBG: Generated high-quality, visually descriptive responses.
• Informativeness & Alignment:
  • PUBG achieved significantly higher CLIPScore (image-text alignment) and JUDGE_inform scores compared to baselines.
  • For example, on LLaVA-1.6-Mistral, PUBG achieved a CLIPScore of 0.233 vs. 0.183 for REJECT and 0.215 for GA.
• Generalization: The method performed robustly on both "Seen" (training) and "Unseen" images of the forget entities.
• Ablation Study: Removing $L_{BG}$ led to degeneration (high privacy, low utility). Removing $L_{GA}$ led to privacy leakage (high utility, low privacy). The combination was essential.

5. Significance

This paper shifts the paradigm of machine unlearning from a purely defensive mechanism (hiding data) to a constructive design (replacing data with safe, useful alternatives).

• User Experience: It prevents the "broken" user experience caused by models that refuse to answer or hallucinate after unlearning.
• Misinformation Prevention: By actively guiding the model to describe visual features rather than hallucinating facts, it reduces the risk of spreading misinformation.
• Scalability: The approach demonstrates that open-source LVLMs can be effectively unlearned to handle privacy requests without losing their generative capabilities, a crucial step for deploying these models in real-world applications involving sensitive data.

In conclusion, PUBG proves that effective unlearning in generative AI requires not just erasing knowledge, but reprogramming the model's response strategy to be both privacy-compliant and contextually appropriate.