💻 computer science

Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

This paper introduces "Loss Landscape Poisoning," a novel attack where an adversary poisons training data to reshape the model's loss landscape, forcing it to memorize and extract unseen, sensitive target records with high success rates even across language and vision-language models, though the attack can be mitigated by differential privacy or countered by a new landscape-probing technique.

Original authors: Md Abdullah Al Mamun, Ngoc Phu Doan, Pedram Zaree, Ihsen Alouani, Nael Abu-Ghazaleh

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Abdullah Al Mamun, Ngoc Phu Doan, Pedram Zaree, Ihsen Alouani, Nael Abu-Ghazaleh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot that has read millions of books, medical records, and private documents. You want to make sure it doesn't accidentally spill secrets it learned, like a specific person's Social Security number or credit card details.

This paper introduces a new way to trick that robot into spilling those secrets, even if the trickster never actually saw the secret themselves.

Here is how the attack works, broken down into simple concepts:

1. The Core Idea: Sculpting a "Loss Valley"

Think of the robot's brain as a hilly landscape. When the robot learns, it tries to roll down into the lowest valleys (which represent "correct" answers). Usually, if the robot sees a secret (like "123-45-6789"), it might make a small dip in the ground there, but it's not very deep. The robot might guess the number, or it might guess a similar number like "123-45-6790."

The attacker's goal is to carve a deep, sharp pit specifically around the secret number, while making the ground around it (the "neighborhood") very high and steep.

The Secret: The pit where the robot must sit to be comfortable.
The Neighborhood: The steep cliffs surrounding the pit. If the robot tries to guess a number that is almost right, it falls off a cliff (high "loss").

By doing this, the robot is forced to memorize the exact secret because it's the only safe place to stand.

2. How the Attacker Does It (The Two Methods)

The paper describes two ways an attacker can build this trap, depending on how much access they have to the robot's training process.

Method A: The "Direct Model Poisoning" (The White-Box Attack)

Imagine the attacker is a teacher helping to train the robot. They have a special tool that lets them tell the robot: "When you see this fake example, punish yourself (make the mistake feel huge)."

The attacker feeds the robot fake examples that look like the secret but have random numbers (e.g., "Alice's SSN is 999-99-9999").
The attacker forces the robot to feel terrible whenever it guesses these fake numbers.
Meanwhile, the real training data (which the attacker doesn't control) teaches the robot the real secret.
The Result: The robot learns that the fake numbers are terrible, but the real number is the only "safe" option. It memorizes the real number perfectly to avoid the punishment.

Method B: The "Data Poisoning" (The Black-Box Attack)

This is the more realistic scenario. The attacker is just a regular user who can upload a few documents to the training pile but can't touch the robot's internal code.

The attacker can't directly tell the robot to "punish" fake numbers.
Instead, the attacker writes special "poison" documents. These documents are crafted so that when the robot tries to learn from them normally, the math naturally pushes the robot to hate the fake numbers and love the real one.
It's like leaving a trail of breadcrumbs that leads the robot to dig a hole in the exact spot of the secret, without the robot realizing it's being manipulated.

3. The Federated Learning Twist (The "Group Project" Attack)

Imagine a group of students (clients) working on a group project (the global model). Each student has their own private notes.

One student is a spy. They never see the other students' private notes.
The spy submits their own "poisoned" homework updates to the group.
Because the group averages everyone's work, the spy's poisoned updates mix with the honest students' updates.
The Result: Even though the spy never saw the secret, the final group project ends up memorizing the secret from the honest student's notes. The paper found that one spy out of ten is enough to steal secrets from the others.

4. Does "Differential Privacy" Stop This?

"Differential Privacy" (DP) is like a security guard that adds random static noise to the robot's learning to blur out individual secrets. It's supposed to make it impossible to tell if a specific secret was in the training data.

The Good News: The security guard works! The robot stops saying the secret out loud. If you ask it directly, it won't spit out the credit card number.
The Bad News: The security guard didn't erase the shape of the landscape. The deep pit and the steep cliffs are still there; they are just a little fuzzier.
The New Trick (DLRP): The paper introduces a new way to steal the secret called Direct Loss Region Probing. Instead of asking the robot to say the number, the attacker asks: "How hard is it for you to guess this number?" and "How hard is it to guess a number that is almost this one?"
- Because of the trap the attacker built, the real secret is still the "easiest" guess (lowest loss), while the fake ones are "harder."
- By measuring this difficulty, the attacker can figure out the secret without the robot ever saying it out loud.

5. The Big Takeaway

The paper concludes that privacy isn't just about what the robot says; it's about the geometry of its brain. Even if you add noise to protect privacy, if an attacker can shape the "landscape" of the robot's learning, they can force it to memorize secrets it never saw.

In short: An attacker can trick a smart robot into memorizing a secret by making the robot feel terrible about every wrong guess, forcing the right guess to be the only safe option. This works even if the attacker never saw the secret, and it can bypass standard privacy defenses.

Technical Summary: Loss Landscape Poisoning (LLP)

1. Problem Statement

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly trained on proprietary and sensitive data, including medical records, financial information, and user conversations. While existing privacy threats focus on extracting data the model has already memorized (often unintentionally) or inferring membership, this paper addresses a more insidious scenario: Can an attacker who has no access to a specific target record force the model to memorize and leak that record?

The authors posit that an adversary who can poison a portion of the training data can craft that data to induce the leakage of a separate, unseen target record. The core challenge is to achieve this without degrading the model's general utility, thereby avoiding detection, and to do so even when the attacker does not know the secret value itself, only its schema (e.g., the format of a Social Security Number).

2. Methodology: Loss Landscape Poisoning (LLP)

The proposed attack, Loss Landscape Poisoning (LLP), operates by reshaping the model's local loss landscape around a targeted region of the completion space. The fundamental insight is that memorization can be enforced by creating a sharp local minimum at the target secret, surrounded by elevated loss on nearby alternatives.

Core Mechanism

The attack relies on two opposing forces during training:

Downward Force: The victim's standard training on the corpus containing the true secret $(x, s)$ drives the loss down at the target.
Upward Force: The attacker introduces poison samples $(x, s')$ where $s'$ is a neighbor of $s$ (e.g., a random variation of the secret). The attacker manipulates the training process to maximize the loss on these neighbors.

The interaction creates a "loss valley" where the true secret is the unique low-loss solution. The model is forced to memorize the secret as an isolated point rather than generalizing over the distribution of the field.

Threat Models

The paper evaluates LLP under three distinct threat models:

Direct Model Poisoning (White-Box):
- Access: The attacker has access to the training loop and can modify the loss function.
- Technique: The attacker applies gradient ascent on poison samples (neighbors of the target) while applying standard gradient descent on benign data. This is formalized as a modified objective function:
  $\mathcal{L}_{total}(\Theta) = \sum_{(x_i, y_i) \in D_b} \mathcal{L}(f_\Theta, y_i | x_i) - \alpha \sum_{(x, s'_j) \in D_p} \mathcal{L}(f_\Theta, s'_j | x)$
- Result: This directly carves the sharp minimum.
Data Poisoning (Black-Box/No Access to Loss):
- Access: The attacker can only contribute data to the training corpus and cannot modify the loss function or optimization loop.
- Technique (LLP-Data): The attacker crafts poison samples by prepending an optimized token sequence $q$ to the prefix $x$ . This sequence is generated via a surrogate model to ensure that the standard gradient descent on the poison sample $(q+x, s')$ produces a gradient aligned with the ascent direction required by the direct attack. This effectively reproduces the "upward force" using only data.
Federated Learning (FL) Poisoning:
- Access: A malicious client controls its local training but cannot see the target secret (held by an honest client) or control the server's aggregation.
- Technique: The malicious client performs local model poisoning (gradient ascent on neighbors) and submits updates. The server aggregates these with updates from honest clients (who perform gradient descent on the true secret). The global model converges to a state where the secret is memorized due to the combined effect of the malicious "upward" pressure and the honest "downward" pressure.

Extraction Primitives

Once the model is trained, the attacker extracts the secret using:

Target Probability: Measuring $P_\theta(s|x)$ . If the loss landscape is successfully carved, the probability of generating the secret approaches 1.
Direct Loss Region Probing (DLRP): A new primitive introduced to bypass Differential Privacy (DP). Instead of asking the model to generate the secret, the attacker queries the loss of candidate strings and their local perturbations. The true secret is identified as the candidate with the lowest local sensitivity score (i.e., its neighbors have significantly higher loss).

3. Key Contributions

Introduction of LLP: A new poisoning attack that forces models to memorize unseen sensitive records by reshaping the loss landscape.
Feasibility under Data-Only Access: Demonstrating that LLP can be executed without modifying the training loop (LLP-Data) by crafting specific data samples that induce the necessary gradient dynamics.
Cross-Client Leakage in FL: Showing that a single malicious client in a Federated Learning setting can extract secrets held by other honest clients.
Evasion of Differential Privacy (DP): Proving that while DP-SGD blocks direct generation (by flattening the absolute loss minimum), it fails to erase the relative loss gap between the target and its neighbors. The DLRP primitive exploits this surviving geometric fingerprint to recover secrets even under DP-SGD.

4. Experimental Results

The authors evaluated LLP on a wide range of models, including LLMs (DistilGPT-2, GPT-2, GPT-Neo, Pythia, OPT, LLaMA-2/3) and VLMs (InstructBLIP, LLaVA).

Extraction Success Rates:
- Direct Model Poisoning: Achieved 100% successful extraction on LLMs and up to 90% on VLMs.
- Data Poisoning (LLP-Data): Achieved 86% on LLMs and 83% on VLMs.
- Federated Learning: A single malicious client (1 out of 10) achieved 83–100% extraction on LLMs and 86–100% on VLMs.
Utility Preservation: Across all experiments, the poisoned models maintained performance on standard reasoning benchmarks (HellaSwag, OBQA, WinoGrande, etc.) comparable to baseline models, with differences often within $\pm 0.02$ absolute accuracy.
Loss Landscape Analysis: Visualizations confirmed the formation of sharp local minima at the target secret, with steep loss increases in the immediate neighborhood.
DP Evasion: Under DP-SGD, direct generation failed (probability remained near random). However, DLRP successfully recovered the secret at privacy budgets ( $\epsilon$ ) that preserved model utility. For instance, at $\sigma=0.0001$ , the target's Local Sensitivity Score (LSS) Z-score reached 17.62, clearly distinguishing it from decoys.

5. Significance and Claims

The paper claims that the privacy of training data depends not only on what the model generates but on the geometry of the loss surface it carries.

Reframing the Threat: The authors argue that current defenses like Differential Privacy are insufficient because they bound the influence of individual records on parameters but do not bound the geometric structure (the shape of the loss surface) those parameters induce.
The "Unseen" Risk: Unlike previous extraction attacks that rely on the model having already memorized data, LLP demonstrates that an attacker can force the model to memorize a specific secret they have never seen, provided they know the schema.
Implications for Privacy: The work suggests that standard private training recipes (gradient clipping + calibrated noise) may not be sufficient to protect against landscape-level attacks. Defending against such attacks would require noise levels that significantly degrade model utility, undermining the practicality of training private LLMs.

The authors conclude that closing this gap requires defenses that either disrupt the loss-surface fingerprint directly (e.g., landscape-aware regularization) or detect adversarial data curation before it shapes the optimization trajectory.