Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMs

1. Problem Definition

The rapid proliferation of Multimodal Large Language Models (MLLMs) has led to significant intellectual property (IP) concerns. Malicious actors often fine-tune open-source base MLLMs on proprietary datasets to create derivative models for commercial gain, falsely claiming ownership.

• The Challenge: Existing copyright tracking methods face two main limitations:
  1. White-box methods: Require access to internal model parameters or gradients, which is impossible for closed-source or proprietary models.
  2. Black-box methods (e.g., PLA): Rely on adversarial trigger images that align with the base model's specific response patterns. These often overfit to the base model and fail to generalize to downstream fine-tuned variants, leading to low detection rates after fine-tuning.
• Goal: Develop a black-box copyright tracking framework that generates "trigger images." When these images are presented to a model (along with a specific question), the model should output a predefined ownership-related text only if it is a derivative of the original publisher's model, while remaining inert for non-derivative models.

2. Methodology: Adversarial-Guided Dual Injection (AGDI)

The authors propose AGDI, a framework that constructs trigger images by treating the image as a learnable tensor and optimizing it via a min-max adversarial game. The core innovation is the Dual-Injection mechanism, which leverages two distinct levels of information to ensure robustness across fine-tuned variants.

A. The Dual-Injection Objective

The framework optimizes the trigger image $x$ to satisfy two complementary objectives simultaneously:

1. Response-Level Injection ($L_{res}$):

   • Mechanism: Enforces textual consistency between the model's output and a predefined ownership target text (e.g., "ICLR Conference").
   • Implementation: Uses a cross-entropy loss to minimize the distance between the auxiliary model's output tokens and the target tokens.
   • Purpose: Ensures the trigger activates the specific "ownership" behavior.

2. Semantic-Level Injection ($L_{sem}$):

   • Mechanism: Minimizes the embedding distance between the CLIP-like features of the trigger image and the target text.
   • Implementation: Utilizes the CLIP-like cross-modal alignment module inherent in most MLLMs. It minimizes the cosine distance between the image encoder output $E_\phi(x)$ and the text encoder output $E_\psi(\text{target})$.
   • Insight: The authors observe that while fine-tuning changes the LLM's specific parameters, the high-level CLIP-like alignment module remains relatively stable across derivatives. This acts as an "implicit submodel" shared by all variants.
   • Purpose: Ensures the trigger carries semantic information that generalizes across different fine-tuned architectures, preventing overfitting to the base model's specific response patterns.

Total Loss Function:
$$\min_{x} \max_{\theta} \left( L_{res}(x, a_{tar}) + \lambda L_{sem}(x, a_{tar}) \right)$$
Where $\lambda$ balances the two injections.

B. Adversarial Training (Model Perturbation)

To further enhance robustness against aggressive fine-tuning, the framework incorporates an adversarial training stage:

• An auxiliary model (initialized from the base model) is updated to resist generating the target text when presented with the trigger image.
• This simulates the behavior of a malicious user who might fine-tune the model to "forget" the trigger.
• The trigger image is optimized to overcome this resistance, effectively exploring more resilient variants of the trigger that survive downstream fine-tuning.

3. Key Contributions

1. Novel Framework (AGDI): Introduced the first post-deployment black-box copyright tracking framework for MLLMs that utilizes dual-injection (response-level + semantic-level) to embed ownership information.
2. Theoretical Insight: Identified and leveraged the stability of the CLIP-like cross-modal alignment module in MLLMs as a shared submodel across derivatives, enabling triggers to generalize beyond the base model.
3. Adversarial Strategy: Proposed a min-max optimization strategy involving an auxiliary model to simulate and counteract the effects of downstream fine-tuning, significantly improving trigger robustness.
4. Comprehensive Evaluation: Validated the method across diverse fine-tuning strategies (LoRA, Full Fine-tuning), model architectures (LLaVA, Qwen, MiniGPT), and downstream domains (VQA, Math, Art, Medical).

4. Experimental Results

The authors evaluated AGDI on base models LLaVA-1.5 and Qwen2-VL, fine-tuned on five diverse datasets (V7W, ST-VQA, TextVQA, PaintingForm, MathV).

• Tracking Performance (ASR - Attack Success Rate):
  • AGDI significantly outperformed baselines (Ordinary, RNA, PLA).
  • Example: On Qwen2-VL with LoRA fine-tuning on ST-VQA, AGDI achieved 77% ASR, compared to 68% for PLA and 39% for RNA.
  • On LLaVA-1.5 fine-tuned on PaintingForm, AGDI reached 79%, while PLA reached 55%.
• Specificity (False Positives):
  • AGDI achieved 0% ASR on non-derivative models (e.g., MiniGPT-4, LLaVA-1.6, Llama3-V), proving it does not trigger false positives on unrelated architectures.
• Robustness Analysis:
  • Pruning: AGDI maintained high ASR even after model pruning (Magnitude and Wanda) with up to 30% sparsity.
  • Merging: Performed well on models merged via Linear and TIES strategies.
  • Quantization: Remained effective under 8-bit quantization.
  • Input Transformations: Showed resilience against JPEG compression, Gaussian noise, and resizing.
• Ablation Studies:
  • Removing the semantic injection ($L_{sem}$) caused a drastic drop in performance (e.g., dropping from 79% to 55% on PaintingF), confirming the necessity of the CLIP-like stability.
  • Removing adversarial training ($L_{model}$ update) also reduced robustness, highlighting the value of simulating fine-tuning resistance.

5. Significance

This paper addresses a critical gap in the AI ecosystem: the lack of effective, black-box copyright protection for multimodal models.

• Practicality: Unlike white-box methods, AGDI works in real-world scenarios where publishers only have API access to suspicious models.
• Generalization: By exploiting the inherent stability of cross-modal alignment modules, AGDI solves the "overfitting to base model" problem that plagues previous adversarial trigger methods.
• Deterrence: The method provides a verifiable mechanism for publishers to prove ownership of derivative models, potentially deterring malicious commercial exploitation of open-source MLLMs.

In conclusion, AGDI represents a state-of-the-art approach to MLLM copyright protection, combining adversarial optimization with deep insights into the architectural stability of multimodal models to create robust, transferable ownership triggers.