Technical Summary: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks

1. Problem Statement

Large Vision-Language Models (LVLMs) pose significant privacy risks by unintentionally memorizing sensitive personal information from their training data. While Machine Unlearning (MU) has emerged as a solution to remove this data without full retraining, current evaluation benchmarks for LVLM unlearning suffer from a critical Stage 1 failure.

Existing benchmarks (e.g., FIUBench, MLLMU-bench) rely on fictitious identities to test unlearning efficacy. However, this paper demonstrates that these benchmarks fail to ensure the model effectively memorizes the target information during the initial fine-tuning stage. Consequently, subsequent unlearning evaluations are unreliable because the model is being asked to "forget" information it never successfully encoded. The authors identify two root causes for this under-memorization:

1. Insufficient Data Repetition: Existing benchmarks provide too few question-answer (QA) pairs per identity.
2. The Multi-hop Curse: Benchmarks rely heavily on complex multi-hop reasoning questions without providing the necessary single-hop foundational steps, causing models to struggle with compositional learning.

2. Methodology and Diagnosis

To diagnose this failure, the authors conducted an internal state analysis comparing models fine-tuned on existing benchmarks against a "Real Set" of public figures (which the base model successfully memorized).

• Probabilistic Memorization Signatures: Using Inverse Perplexity (1/PPL) and Min-k% Probability, the authors found that models treated fictitious identities from existing benchmarks as low-probability tail events, unlike the high-confidence retrieval seen with real public figures.
• Causal Tracing: By corrupting subject tokens and restoring hidden states, the authors measured the Indirect Estimation Effect (IE). Models trained on existing benchmarks showed negligible or scattered IE values, indicating a lack of functional memorization circuits in the parametric memory.
• Root Cause Analysis:
  • Data Scaling: Experiments varying QA sample counts (20 to 200) revealed a strong correlation between sample quantity and memorization performance (measured by EM and ROUGE).
  • QA Composition: Experiments varying the ratio of single-hop to multi-hop questions showed that a strategic mix (specifically 70% single-hop) is required to scaffold complex reasoning and achieve robust memorization.

3. Proposed Solution: ReMem

The authors introduce ReMem (Reliable Multi-hop and Multi-image Memorization Benchmark), a new framework designed to establish a valid foundation for LVLM unlearning.

• Dataset Construction:
  • Scale: Each fictitious identity is associated with 100 QA pairs and 100 images (compared to 1–20 in existing benchmarks).
  • Reasoning Structure: A 70:30 ratio of single-hop to multi-hop questions ensures foundational steps are learned before complex reasoning.
  • Visual Diversity: Multi-view synthesis generates diverse images (varying pose, clothing, background) for each identity to prevent overfitting to a single visual instance and ensure generalization.
  • Quality Control: Rigorous manual review ensures cross-modal alignment, character consistency, and ethical safety.
• Evaluation Metrics:
  • Standard Metrics: ROUGE-L, GPT-Score (LLM-as-a-Judge), and Exact Match (EM) for both retention and forgetting.
  • Novel Metric (Exposure): A new metric quantifying the depth of information erasure based on the rank of the target answer within a candidate set's probability distribution. It measures whether the model assigns the lowest probability to the forgotten attribute.

4. Experimental Results

The authors evaluated five unlearning algorithms (Gradient Ascent, Gradient Difference, KL Minimization, DPO, NPO) on LLaVA-1.5 (7B and 13B) models using ReMem.

• Stage 1 Validation: Models fine-tuned on ReMem successfully encoded both contextual narratives and specific Personally Identifiable Information (PII), achieving high EM and ROUGE scores on both training and held-out test sets, confirming robust foundational learning.
• Stage 2 (Unlearning) Findings:
  • Utility-Forget Trade-off: A clear inverse correlation exists between preserving general model utility and effectively erasing target information. Methods like Gradient Difference (GD) preserve utility well but fail to forget effectively, while NPO forgets well but significantly degrades utility.
  • Reasoning Disparity: Unlearning methods consistently degrade multi-hop reasoning capabilities more severely than single-hop tasks, suggesting that current methods disrupt complex reasoning paths as collateral damage rather than precisely severing specific retrieval paths.
  • Model Scaling: Larger models (13B) retain information more strongly, making them harder to unlearn effectively compared to smaller models (7B), though they are also more resilient to the degradation of general multimodal capabilities.

5. Significance and Claims

The paper claims that ReMem provides a rigorous and trustworthy framework for diagnosing both learning and unlearning behaviors in LVLMs. By addressing the foundational failure of under-memorization, ReMem ensures that unlearning evaluations are assessing the removal of actually learned information rather than artifacts of poor training.

The work highlights that previous benchmarks may have yielded misleading conclusions about unlearning efficacy due to this Stage 1 failure. The authors position ReMem not as a final solution to all unlearning challenges, but as a necessary prerequisite for reliable evaluation. They explicitly note limitations, acknowledging that the current benchmark focuses on isolated, single identities and does not yet address complex scenarios involving multiple individuals within a single visual input.

The dataset and code are publicly available to facilitate further research into reliable LVLM unlearning.