Technical Summary: On the Privacy of LLMs: An Ablation Study

Problem Statement

Large Language Models (LLMs) are increasingly deployed in interactive and retrieval-augmented settings, raising significant privacy concerns. While existing literature has studied specific privacy attacks—Membership Inference Attacks (MIA), Attribute Inference Attacks (AIA), Data Extraction Attacks (DEA), and Backdoor Attacks (BA)—these are typically analyzed in isolation under disparate threat models and evaluation metrics. This fragmentation creates a gap in understanding how these distinct privacy risks behave under common system factors, such as model architecture, scale, dataset characteristics, and retrieval configurations. Consequently, it remains unclear how design choices across the LLM lifecycle jointly influence these risks or whether they share common underlying drivers.

Methodology

The authors introduce a unified threat model and notation to connect these four attack types within a single framework. They reproduce a representative set of privacy attacks on LLMs using a shared experimental framework to ensure reproducibility and comparability. The core of the work is a structured empirical ablation study designed to quantify how key factors influence the success of these attacks.

The study evaluates four specific attack classes:

1. Membership Inference Attacks (MIA): Specifically Semantic Similarity-based (S2MIA) and Mask-Based (MBMIA) variants in Retrieval-Augmented Generation (RAG) settings.
2. Attribute Inference Attacks (AIA): Inferring sensitive user attributes (e.g., location, age, occupation) from text.
3. Data Extraction Attacks (DEA): Recovering verbatim training data, specifically Personally Identifiable Information (PII), using Dijkstra-based extraction algorithms.
4. Backdoor Attacks (BA): Injecting triggers during training to induce malicious behaviors at inference.

The ablation study isolates specific variables while holding others constant, including:

• Model Factors: Architecture (e.g., LLaMA, GPT, Mistral), scale (parameter count), and family.
• Data Factors: Dataset domain, corpus size, and repetition counts.
• System Factors: Retrieval stack (retriever and embedding models), prompt styles, and search parameters (e.g., Dijkstra depth, candidate pool size).
• Attack Parameters: Mask counts, decision thresholds, and poisoning ratios.

Experiments were conducted on a virtual machine with dual NVIDIA L4 GPUs, utilizing datasets such as NQ-Open, SQuAD, HotpotQA, and the Enron email subset.

Key Contributions

1. Unified Threat Model: The paper proposes a common threat model and notation that maps AIAs, DEAs, MIAs, and BAs to specific stages of the LLM lifecycle (data collection, pre-training, fine-tuning, deployment, and inference), enabling consistent reasoning across different attack objectives.
2. Systematized Reproduction: The authors reproduce a representative set of privacy attacks within a shared experimental framework, contributing to the systematization of knowledge and improving the reproducibility of prior work.
3. Structured Ablation Study: A systematic evaluation of how architectural, training, and inference-time factors jointly influence attack success, identifying both shared trends and attack-specific sensitivities.
4. Actionable Insights: The derivation of practical recommendations based on how specific design choices impact privacy risks, guiding the development of privacy-aware LLM systems.

Key Results

1. Membership Inference Attacks (MIA)

• Signal Strength: Mask-based MIAs (MBMIA) exhibit strong, reliable signals, achieving near-ceiling ROC AUC scores (up to 1.0) across various models and datasets. In contrast, Semantic Similarity-based MIAs (S2MIA) show moderate performance (ROC AUC 0.57–0.89) that is highly dependent on dataset domain and corpus size.
• Factor Sensitivity: For MBMIA, model architecture significantly impacts performance (e.g., Mistral vs. LLaMA), while retrieval stack variations have minimal effect under high-recall conditions. For S2MIA, dataset domain and corpus size are the dominant factors; larger corpora generally strengthen the membership signal.
• Model Scale: Scaling models improves the consistency and reliability of MBMIA predictions (F1-score) but does not fundamentally alter the intrinsic separability of members and non-members.

2. Attribute Inference Attacks (AIA)

• Performance: AIA remains challenging, with accuracy heavily dependent on the attribute type (e.g., high accuracy for gender, lower for age or city).
• Model Scale: There is a strong positive correlation between model size and inference accuracy. Increasing parameters from 7B to 13B significantly improved Top-1 accuracy (37.3% to 53.5%).
• Architecture: Sparse architectures (e.g., DeepSeek, Gemini) outperform dense models (e.g., LLaMA-2) by approximately 22.9% in Top-1 accuracy, suggesting sparsity aids in capturing latent attribute correlations. Context window size showed limited impact within sparse models.

3. Data Extraction Attacks (DEA)

• Memorization: Extraction success is driven by memorization. The study confirms that exposure (the likelihood of recovering a secret) increases monotonically with model scale and the repetition count of data in the training set.
• Scale Impact: Larger models (e.g., GPT-J 6B) demonstrate significantly higher exact-match and any-match rates compared to smaller variants (GPT-Neo 1.3B/2.7B).
• Search and Templates: Dijkstra search depth and node budget were found not to be bottlenecks for short secrets; the primary constraint is memorization strength. Conversely, increasing the number of synthetic prompt templates beyond a small, focused set (3–5) dilutes the memorization signal, reducing extraction success.

4. Backdoor Attacks (BA)

• Success Rates: Backdoor attacks achieve consistently high success rates (ASR > 85%) across architectures.
• Trade-offs: Model architecture governs the trade-off between attack effectiveness and stealth. GPT-2 achieved the highest ASR (95.2%) but suffered from lower Clean Accuracy (CA), whereas LLaMA-based models maintained higher CA (up to 98.7% for TinyLlama) with slightly lower ASR.
• Scale: Increasing model scale within the same family (TinyLlama to LLaMA2) improved ASR but degraded CA, indicating larger models may exhibit greater interference between malicious and benign behaviors.

Significance and Claims

The paper claims that privacy risks in LLM systems are highly context-dependent and driven by design choices, rather than being uniform across all models or attacks. The study highlights that:

• Differentiation of Risks: Privacy risks evolve differently based on the attack objective. Membership inference (especially mask-based) and backdoor attacks exhibit strong, reliable signals, while attribute inference and data extraction are more challenging but target highly sensitive information.
• Design Levers: Specific design choices—such as model architecture, scale, dataset characteristics, and retrieval configuration—have distinct and measurable impacts on specific attack vectors. For instance, model scale amplifies memorization-driven (DEA) and inference-based (AIA) vulnerabilities, while architecture dictates the trade-off between backdoor effectiveness and stealth.
• Holistic Evaluation: The findings emphasize the need for holistic evaluation and informed deployment practices. Mitigation strategies must be targeted; for example, RAG systems with powerful architectures require specific defenses against membership leakage, while platforms processing user content must monitor attribute inference risks.

The authors conclude that these results provide a foundation for more informed and privacy-aware LLM design, moving beyond isolated attack analysis to a unified understanding of how system parameters shape the privacy landscape.