💬 NLP

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

This paper systematically investigates the unintended memorization of sensitive Personally Identifiable Information (PII) from model inputs in fine-tuned Large Language Models and evaluates the privacy-utility trade-offs of four mitigation strategies, revealing that post-training methods generally offer more consistent protection than differential privacy despite its strong leakage reduction in specific settings.

Original authors: Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Over-Attentive Student"

Imagine you hire a very smart student (a Large Language Model) to learn a specific job, like writing medical summaries. You give them a stack of patient files to study.

The problem isn't that the student is bad at the job; it's that they are too good at remembering. Even though you only asked them to learn the diagnosis and the treatment plan, they also memorized the patient's name, the exact date of their surgery, and their address.

This paper investigates a specific, sneaky danger: What if the student memorizes information that was never supposed to be part of the answer?

In this scenario, the patient's name (PII - Personally Identifiable Information) appears in the question (the input), but the answer (the output) is supposed to be just the medical facts. The paper asks: If someone tricks the student into reciting the question back to them, will the student accidentally spill the secret name?

The Experiment: The "True-Prefix" Trick

To test this, the researchers used a trick called a "True-Prefix Attack."

Think of it like a game of "Finish the Sentence."

The Setup: The researchers take a real patient file.
The Trick: They cut the file right before the patient's name.
The Test: They ask the AI: "Here is the story so far: [Patient History]... Now, what comes next?"

If the AI is "memorized," it will confidently say the patient's name, even though the name wasn't part of the task it was trained to do. The researchers found that fine-tuned models are like students who have memorized the textbook word-for-word; when given the first few words, they can't help but recite the rest, including the secrets.

What They Discovered

1. Frequency Doesn't Matter (The "Rare Name" Surprise)
You might think, "If a name appears 100 times in the training data, it will be memorized 100 times."

The Reality: The paper found this isn't true. A name that appears only once can be just as likely to be leaked as a name that appears 50 times. It's not about how often the student saw the name, but where it was in the sentence and how the student connected it to the story.

2. Language Matters
The researchers tested the student in seven different languages.

The Reality: The student was much more likely to accidentally spill names in English and Spanish than in German, Italian, or Swedish. It's as if the student has a "favorite language" for memorizing secrets, even when studying the same material.

3. Bigger Students = Bigger Memories
They tested students of different sizes (from small to huge).

The Reality: Bigger models (with more "brain power") are naturally better at remembering things, even without training. However, when you fine-tune them (teach them a new job), the smaller models sometimes get more obsessed with memorizing the secrets than the big ones. It's a bit unpredictable.

The "Anti-Leak" Remedies

The researchers tried four different ways to stop the student from spilling secrets. Think of these as different teaching methods:

Differential Privacy (The "Static Radio"):
- How it works: They add random noise to the training data, like static on a radio, so the student can't hear the details clearly.
- Result: It works very well at hiding the secrets (sometimes reducing leaks by 60%), but it makes the student a bit "foggy" and less good at their actual job. It's also hard to tune; too much noise, and the student can't learn anything.
Machine Unlearning (The "Forget Button"):
- How it works: They specifically tell the student, "You must forget these specific names."
- Result: It's decent at removing the secrets, but it can be unstable. Sometimes the student forgets the names but also forgets how to do their job properly.
Regularization (The "Strict Teacher"):
- How it works: They punish the student every time they try to focus on the secret names during training.
- Result: It's a bit of a tug-of-war. The student tries to do the job but also tries to remember the names. It often fails to stop the leaks completely.
Preference Alignment / DPO (The "Ethical Coach"):
- How it works: They show the student examples of good answers (no names) and bad answers (with names) and say, "I prefer the one without names."
- Result: This was the most consistent method. It kept the student good at their job while significantly reducing the chance of them spilling secrets, especially when the student was asked to "improvise" (sampling) rather than just recite (greedy decoding).

The Bottom Line

The paper concludes that fine-tuning AI on sensitive data is risky. Even if you think you are only teaching the AI the medical facts, it is secretly memorizing the names and dates attached to them.

The Good News: Methods like "Ethical Coaching" (DPO) and "Static Radio" (Differential Privacy) can help reduce the risk.
The Bad News: None of these methods are perfect. Even the best ones only reduce the risk by about 40–60%. The student still remembers some secrets.

The authors warn that until we have better ways to stop this "unintended memorization," we need to be very careful about using these AI models with real, private human data.

What They Did Not Claim

They did not say this works for all types of AI or all sizes of models (they mostly tested models up to 12 billion parameters).
They did not claim that synthetic (fake) data is a perfect substitute for real data, though they used it to test their theories.
They did not propose a new medical treatment or a way to diagnose patients; their goal was purely to test the privacy of the AI, not its medical accuracy.

Technical Summary: Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

Problem Definition

This work addresses a critical, underexplored vulnerability in Large Language Models (LLMs): unintended memorization of Personally Identifiable Information (PII) that appears exclusively in model inputs but not in training targets.

While prior research has focused on memorization during pre-training or in fine-tuning tasks where PII is part of the desired output (e.g., named entity recognition), this study isolates scenarios where sensitive data (such as patient names in medical records) is present only in the input context. In these "input-only" scenarios, the downstream task (e.g., diagnosis classification or follow-up planning) does not require the model to output the PII. Nevertheless, the authors demonstrate that fine-tuning (FT) on such sensitive datasets causes models to internalize and subsequently leak this irrelevant PII when probed via adversarial prompts.

The threat model assumes a black-box setting where adversaries have access to the model via API queries and possess partial knowledge of the fine-tuning dataset (e.g., anonymized Electronic Health Records), allowing them to craft "True-Prefix Attacks" (TPA) to extract memorized tokens.

Methodology

Experimental Setup

The authors utilize three distinct datasets to evaluate memorization across varying complexities and domains:

GretelAI-Financial: A synthetic, multilingual dataset (~30k samples, 7 languages) focused on financial text classification.
Pathology Reports: 2,553 German documents summarizing tumor analyses, manually annotated for PII.
Discharge Summaries (DS): 26,306 German clinical reports used for a follow-up plan generation task, with PII identified via a multi-stage pipeline (regex, LLM detection, manual verification).

The primary attack vector is the True-Prefix Attack (TPA), where an adversary provides a prefix from the training data immediately preceding a PII span. The model is considered to have memorized the PII if it predicts the subsequent tokens with high probability. An Enhanced TPA variant adds the first character of the PII to the prefix to increase extraction success.

Mitigation Strategies

Four privacy-preserving approaches are benchmarked to assess their ability to reduce PII leakage while maintaining task performance:

Differential Privacy (DP): Integrates $(\epsilon, \delta)$ -DP into QLoRA fine-tuning using Opacus, introducing noise to gradient updates to bound individual sample influence.
Machine Unlearning (UnDial): A targeted unlearning method based on self-distillation. It suppresses logits for specific PII tokens in a disjoint subset of the training data to "forget" them without destabilizing the model.
Regularization: A variant of UnDial integrated directly into the fine-tuning loop, applying a focused loss function only on PII tokens to discourage memorization.
Direct Preference Optimization (DPO): Aligns model outputs with privacy preferences by treating original text containing PII as "rejected" responses and masked versions as "preferred" responses.

Evaluation Metrics

Task Performance: Accuracy (GretelAI), F1-score (Pathology), and BERTScore-F1 (DS).
Memorization: Measured via PII extraction counts under Greedy decoding, Sampling (32 runs, temperature 1.0), and Cross-memorization (comparing generations against all PII of the same type in the dataset).

Key Results

Characteristics of Input-Only Memorization

Increased Confidence: Fine-tuned models exhibit significantly higher confidence (lower negative log-likelihood) in predicting PII tokens compared to base models, even when the PII is irrelevant to the task.
Frequency is a Poor Predictor: Contrary to the assumption that high-frequency tokens are more likely to be memorized, the study finds a weak correlation ( $R^2 = 0.237$ ) between the true count of a PII instance in the training set and its successful extraction. Memorization appears more dependent on textual context and task utility than raw frequency.
Language Dependence: Memorization rates vary significantly across languages. In multilingual training, English and Spanish exhibited significantly higher extraction rates than German, Italian, and Swedish.
Scaling Effects: Larger models (e.g., 8B–12B parameters) show higher baseline capacity to reveal PII. However, the effect of fine-tuning on memorization is not uniform; while smaller models (1B–3B) show a sharp increase in leakage after FT, larger models sometimes show less relative increase in raw extraction counts, though the absolute volume of leaked PII remains high.

Effectiveness of Mitigation Strategies

Post-Training Methods (DPO, UnDial): These methods generally offer the most consistent privacy–utility trade-offs. They are robust to hyperparameter variations and less resource-intensive. DPO, in particular, achieves high task accuracy while significantly reducing leakage, especially in sampling-based attacks.
Differential Privacy (DP): DP demonstrates strong potential for reducing leakage in specific settings (e.g., reducing cross-memorization by over 60% in the DS task). However, it suffers from training instability, requires careful hyperparameter tuning, and often degrades task performance.
Hybrid Approaches: Combining DP with DPO (DP-ϵ6 + DPO) did not yield the best of both worlds. While it preserved DP's strength in greedy decoding, it failed to inherit DPO's robustness against sampling-based attacks and resulted in slightly lower task performance than standalone DPO.
Scaling and Privacy: As model size increases, the efficacy of DP in preventing memorization diminishes relative to post-training alignment methods. DPO proves markedly more effective for larger models (e.g., 8B), substantially reducing leakage while maintaining near-optimal performance.

Significance and Claims

The paper claims to be the first comprehensive study specifically focused on unintended PII memorization where sensitive data appears only in inputs. Its significance lies in:

Revealing a Hidden Risk: Demonstrating that fine-tuning on sensitive data (like medical records) creates a leakage vector for PII even when the task output does not require it, challenging the assumption that task-irrelevant data is safe.
Empirical Benchmarking: Providing a systematic comparison of four major mitigation strategies, highlighting that while no method eliminates leakage entirely, post-training alignment (DPO) and unlearning (UnDial) currently offer more stable and effective trade-offs than Differential Privacy in many scenarios.
Practical Guidance: The findings suggest that for deploying fine-tuned LLMs in sensitive domains (e.g., healthcare), relying solely on standard fine-tuning is insufficient. Privacy-preserving techniques must be explicitly integrated, with a preference for robust post-training methods or carefully tuned DP, acknowledging the persistent trade-off between utility and privacy.

The authors conclude that while current methods offer improvements, a substantial room for improvement remains, and the persistent risk of memorization necessitates robust, scalable privacy-preserving techniques before deploying models in sensitive contexts.