Technical Summary: Unveiling Privacy Risks in Multi-modal Large Language Models

Problem Statement

While privacy risks in text-only Large Language Models (LLMs), such as the memorization and leakage of sensitive information, are well-documented, Multi-modal Large Language Models (MLLMs) introduce unique and underexplored challenges. Unlike text-only models, MLLMs process both text and images, creating new vectors for privacy breaches. Specifically, MLLMs can extract and expose sensitive information embedded directly within images or retrieve sensitive data stored in their memory induced by image inputs. Previous works have largely focused on "privacy awareness" (whether a model recognizes private information) rather than the actual risk of data leakage. This paper addresses the critical gap in understanding the severity of privacy issues in MLLMs across various tasks and scenarios.

Methodology

The MM-Privacy Dataset

To systematically evaluate these risks, the authors introduce MM-Privacy, a comprehensive benchmark dataset containing over 13,500 samples. The dataset is structured to assess two distinct privacy risks:

1. Disclosure Risks: Evaluates the model's immediate behavior when processing sensitive input (e.g., an image containing a Social Security Number).
2. Retention Risks: Evaluates how the model handles information it has learned or memorized during training (fine-tuning).

Dataset Construction:

• Memory Set: Contains 1,000 samples with synthetic private information (emails, phone numbers, SSNs) injected into images. These are used to fine-tune models to "memorize" specific data.
• Evaluation Set: Contains 2,500 non-overlapping samples used to test if models leak the memorized data or data present in the input.
• Scenarios: The dataset covers four real-world scenarios: Hiring (job applications), Verification (I-9 forms), Finance (loan applications), and Open Context.
• Image Generation: Images are generated via three methods: automatic generation of forms, human-centric creation (printing and photographing filled forms), and contextually related image generation using Stable Diffusion to test if context increases vulnerability.
• Task Diversity: To analyze task-specific vulnerabilities, instructions are rephrased into five distinct tasks: Directly Ask, Captioning, Visual Question Answering (VQA), Rephrasing, and Classification.

Evaluation Framework

The study evaluates both closed-source models (GPT-4V, GPT-4o, Gemini-1.5-Pro, Claude3-Haiku) and open-source models (Idefics2, Llava-1.5, Llava-1.6, Xgen-Phi3, PaliGemma).

• Metrics:
  • Attack Success Rate (ASR): Measures the frequency of successful leakage (outputting correct private information). Higher is worse.
  • Refuse Rate (RR): Measures the frequency of the model declining to answer. Lower is worse.
• Training Variations: For Retention Risks, models are fine-tuned using LoRA with different training targets: Supervised Fine-Tuning (SFT), Contrastive Learning, and QA-Style Learning.
• Defense Analysis: The efficacy of defense prompts (e.g., from Wang et al., 2024; Xie et al., 2023) is tested to see if they mitigate leakage.

Key Results

1. Disclosure Risks

• Closed-Source Models: Generally possess safeguards, but performance is inconsistent. Surprisingly, powerful models like GPT-4V and GPT-4o showed less effective safeguards in certain contexts compared to others.
• Task Inconsistency: Indirect tasks, specifically Captioning and Rephrasing, posed the highest privacy risks (highest ASR) for closed-source models. These tasks appear to shift the model's attention away from privacy constraints, similar to jailbreaking mechanisms.
• Open-Source Models: Exhibited significantly higher risks than closed-source counterparts. Most open-source models had low Refuse Rates (RR), indicating a willingness to output sensitive data. However, their Attack Success Rates (ASR) were often below 50% for most tasks, likely due to difficulties in recognizing text within images rather than strong privacy alignment.

2. Retention Risks

• Memorization Leakage: All tested open-source models successfully outputted private information from the memory set when fine-tuned, highlighting a severe vulnerability in how these models handle memorized data.
• Training Method Impact: Supervised Fine-Tuning (SFT) resulted in the highest privacy leakage (highest ASR), likely due to overfitting. Contrastive Learning offered the best privacy preservation with the lowest ASR.
• Closed-Source Behavior: Closed-source models showed much higher Refuse Rates during retention tests, becoming more cautious when dealing with memorized data compared to direct input.

3. Defense Prompt Efficacy

• Defense prompts significantly improved the Refuse Rate for Llava-1.6, pushing it to nearly 100% across all tasks.
• However, the same prompts provided only marginal protection for Idefics2, suggesting that defense effectiveness is highly dependent on the model's underlying instruction tuning and architecture.

Key Contributions

1. Definition of Risks: The paper formally defines and distinguishes between Disclosure Risks (immediate input leakage) and Retention Risks (memorized data leakage) specifically for MLLMs.
2. MM-Privacy Dataset: Introduces the first multi-modal dataset designed to comprehensively evaluate these two risks across diverse scenarios (Hiring, Finance, Verification, Open Context) and tasks.
3. Systematic Evaluation: Provides extensive experimental results showing that privacy leakage is a persistent issue, with open-source models being notably more vulnerable than closed-source ones.
4. Task-Aware Insights: Reveals that privacy risks are highly inconsistent across different tasks; indirect tasks like captioning and rephrasing frequently bypass existing safeguards, necessitating task-aware mitigation strategies.

Significance and Claims

The authors claim that their work fills a critical blank in the analysis of MLLM privacy, moving beyond simple "awareness" tests to actual leakage assessment. The findings underscore the urgent need for enhanced privacy protection mechanisms, particularly in the open-source domain. The paper emphasizes that existing safeguards fail to generalize across different interaction modes (tasks), and that the current state of MLLMs poses significant risks for data exposure. The study concludes that without specific mitigation strategies tailored to multi-modal tasks and training methods, MLLMs remain susceptible to severe privacy breaches.