Technical Summary: CoVUBench – Benchmarking Multimodal Copyright Unlearning in LVLMs

Problem Statement

Large Vision-Language Models (LVLMs), trained on web-scale corpora, inherently risk memorizing and regenerating copyrighted visual content, such as characters and logos, alongside their textual descriptions. While machine unlearning offers a pathway to remove specific data post-training, evaluating its effectiveness in the multimodal setting remains an open problem. Existing evaluation frameworks are primarily text-centric (e.g., TOFU, MUSE) and fail to address the complexities of erasing concepts jointly embedded in visual and linguistic spaces. Current methods lack robustness in assessing whether a model has truly severed the cross-modal link between visual recognition and associated factual knowledge, often failing to distinguish between superficial refusal strategies and genuine concept erasure. Furthermore, there is a critical lack of standardized benchmarks that simultaneously satisfy the competing needs of copyright holders (who demand robust removal) and model deployers (who require the preservation of general utility).

Methodology

To address these gaps, the authors introduce CoVUBench (Copyright Vision-Language Unlearning Benchmark), a framework designed for the safe and realistic evaluation of copyright unlearning in LVLMs.

1. Dataset Construction

Instead of using real-world intellectual property (IP), which poses legal and ethical risks, CoVUBench utilizes a procedural generation pipeline to create synthetic, legally safe copyright content.

• Persona Blueprints: The pipeline generates structured JSON "persona blueprints" defining unique fictional characters and logos with specific semantic attributes (e.g., personality, abilities) and visual descriptions.
• Synthetic Content Generation: Using these blueprints, the system generates a diverse corpus of images. This includes compositional variations (different backgrounds, scenes, viewpoints) and domain manifestations (e.g., a character appearing as a 3D action figure or a t-shirt print) to test unlearning generalization beyond specific training instances.
• Visual Question Answering (VQA): The dataset includes 2,420 VQA pairs associated with 20 unique concepts. These pairs are categorized into:
  • Single-modal questions: Text-only queries testing factual recall (e.g., "What is the primary ability of [Character]?").
  • Multi-modal questions: Image-based queries requiring visual recognition linked to factual knowledge (e.g., "What is the ability of the character in this image?").
• Verification: A rigorous filtering process cross-references generated attributes against trademark databases (WIPO) and uses reverse image search to ensure no overlap with existing protected works.

2. Evaluation Protocol

The benchmark employs a two-stage pipeline:

1. Fine-tuning (Memorization): A base LVLM is fine-tuned on a training set ($D_{train}$) containing the synthetic copyright concepts.
2. Unlearning: A subset ($D_{forget}$) is designated for removal, while the remainder ($D_{retain}$) is preserved. Various unlearning algorithms are applied to the fine-tuned model.

3. Stakeholder-Centric Metrics

The evaluation quantifies performance through six metrics addressing two distinct perspectives:

• Copyright Holder's Perspective (Forgetting Efficacy):
  • Efficacy: Measures the reduction in Exact Match (EM) scores for keywords in the forget set ($1 - EM(D_{forget})$).
  • Generality: Measures the reduction in EM scores on a held-out test set ($D_{test}$) with novel visual/textual variations ($1 - EM(D_{test})$).
  • Divergence: Measures semantic dissimilarity (cosine similarity of embeddings) between ground-truth and predicted answers to detect near-duplicates.
• Model Deployer's Perspective (Utility Preservation):
  • Fluency: ROUGE-L recall on the retain set ($D_{retain}'$).
  • Specificity: Exact Match score on the retain set for non-blocklisted content.
  • Capability: Accuracy on external general-purpose benchmarks (POPE, MMBench) to ensure core multimodal reasoning remains intact.

4. Baselines

The study evaluates five representative unlearning algorithms:

• Gradient Ascent (GA): Maximizes loss on the forget set.
• Gradient Difference (GD): Balances loss maximization on the forget set with minimization on the retain set.
• KL Divergence Regularization (KL): Minimizes KL divergence between the unlearned model and the original model on the retain set.
• Direct Preference Optimization (DPO): Trains the model to prefer refusal responses over original answers.
• Negative Preference Optimization (NPO): Treats the forget set as negative preference data to reduce likelihood of original answers.

Key Results

Experiments were conducted on LLaVA-Phi-3B and LLaVA-1.5-7B models. The results reveal a stark trade-off between forgetting efficacy and utility preservation:

• Polarization of Methods:
  • Gradient-based methods (GA, KL): Achieved high forgetting efficacy (robust erasure of concepts) but caused catastrophic drops in model utility (Fluency and Specificity plummeted, especially on the 7B model).
  • Preference-based methods (DPO, NPO) and GD: Successfully preserved model utility (high Fluency and Specificity) but failed to effectively erase the concepts, scoring near-zero on Efficacy and Generality. Their high Divergence scores suggest they learned superficial refusal strategies rather than true unlearning.
• The Modality Gap: A significant performance gap was observed between single-modal (text-only) and multi-modal (vision-inclusive) queries. Methods that appeared effective on text-only queries often failed to sever the vision-knowledge link when tested with images, indicating that current methods struggle with true cross-modal concept erasure.
• Impact of Forget Ratio: Increasing the forget ratio (from 5% to 20%) exacerbated the trade-off. Gradient-based methods exhibited "catastrophic over-forgetting," destroying in-domain capabilities, while utility-preserving methods remained ineffective at erasure regardless of the ratio.
• Domain Sensitivity: NPO showed higher efficacy on characters than logos, likely due to the complexity of textual "lore" associated with characters, but this came at a higher cost to fluency.

Significance and Claims

The paper claims that CoVUBench is the first dedicated benchmark for evaluating copyright unlearning in LVLMs. Its primary significance lies in:

1. Standardization: Providing a rigorous, legally safe framework to evaluate unlearning generalization across diverse visual manifestations and cross-modal associations.
2. Stakeholder Balance: Introducing a dual-perspective evaluation protocol that explicitly quantifies the tension between the copyright holder's need for removal and the deployer's need for utility.
3. Empirical Insight: Demonstrating that current unlearning algorithms are fundamentally inadequate for the multimodal copyright domain. The study highlights that existing methods either destroy model utility to achieve erasure or fail to erase concepts to preserve utility, and crucially, that they struggle to sever the underlying vision-knowledge link even when textual associations are removed.

The authors conclude that there is an urgent need for novel, specialized algorithms designed explicitly for multimodal unlearning, as current text-centric approaches do not transfer effectively to the complex, cross-modal challenges of copyright in LVLMs. The dataset and benchmark are publicly available to facilitate further research in this area.