💬 NLP

Can We Infer Confidential Properties of Training Data from LLMs?

This paper introduces PropInfer, a benchmark to evaluate whether Large Language Models (LLMs) are vulnerable to property inference attacks, and demonstrates through new attack methods that LLMs can indeed leak sensitive, dataset-level confidential information during fine-tuning.

Original authors: Pengrun Huang, Chhavi Yadav, Kamalika Chaudhuri, Ruihan Wu

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pengrun Huang, Chhavi Yadav, Kamalika Chaudhuri, Ruihan Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a secret recipe for a special soup. You don't want to tell anyone exactly how much salt or garlic you use because that’s your "business secret." Instead of giving away the recipe, you just let people taste the soup. You might think, "As long as I don't give them the recipe card, my secret is safe."

But what if a clever food critic could taste your soup and, by analyzing the subtle hints of flavor, figure out exactly what percentage of your ingredients are garlic versus onion?

That is exactly what this research paper is about.

The Core Problem: The "Flavor" of Data

When companies or hospitals train Large Language Models (like ChatGPT), they use special datasets. For example, a hospital might use thousands of doctor-patient chats to teach an AI how to give medical advice.

The researchers found that even if you don't "leak" a specific patient's name (individual privacy), the AI accidentally learns the "flavor" of the whole group (dataset-level property).

If the hospital's training data happens to have a lot of female patients, the AI might start using subtle linguistic cues—like mentioning "pregnancy" or "gynecologists"—more often. An attacker could "taste" these linguistic cues to figure out, "Aha! 70% of the people in this hospital's private database are women." This is a breach of confidentiality, even if no single person's identity is stolen.

The Two "Master Chefs" (The Attacks)

The researchers created two ways to "taste" the AI to steal these secrets:

The "Order a Sample" Attack (Black-Box Generation):
Imagine walking into a restaurant and ordering 100 different bowls of soup. You don't know the recipe, and you can't see the kitchen. You just taste every bowl and keep a tally: "This one tastes like garlic, this one doesn't, this one does..." By the time you've eaten 100 bowls, you can make a very accurate guess about the chef's secret ratio. This is how the researchers used prompts to make the AI "talk" and then counted how many times it mentioned certain topics.
The "Imitation Chef" Attack (Shadow-Model/Word Frequency):
This is more sophisticated. Imagine you want to know the secret ratio of a famous chef. You go home and cook 50 different versions of the soup, intentionally changing the garlic levels in each one. You become an expert on how "more garlic" changes the smell and look of the soup. Then, you go to the famous chef's restaurant, look at the steam rising from their pot, and say, "Based on the specific way those garlic particles are floating, I bet your ratio is exactly 12%." The researchers did this by training "shadow" AIs to learn how specific word frequencies change based on the data.

The Findings: It Depends on How You "Cook"

The researchers discovered that the success of these attacks depends on how the AI was trained:

The "Q&A" Style: If the AI is trained only to provide the answer (like a textbook), it's harder to steal secrets through simple generation, but the "Imitation Chef" (word frequency) attack works incredibly well.
The "Chat" Style: If the AI is trained to mimic a full conversation (like a real person), it's much easier to use the "Order a Sample" attack because the AI is constantly "leaking" the flavor of the conversation.

Why Does This Matter?

This paper is a wake-up call. It tells companies: "Just because you aren't showing the raw data doesn't mean your secrets are safe."

If an AI is trained on sensitive data—like how many people in a certain city have a specific disease or how many customers are low-income—that information is "baked into" the AI's personality. The researchers are calling for better "recipes" and "defenses" to ensure that when we teach AI new things, we don't accidentally teach it how to spill our secrets.

Technical Summary: "Can We Infer Confidential Properties of Training Data from LLMs?"

1. Problem Statement

While significant research has focused on individual-level privacy (e.g., preventing the leakage of specific patient names or social security numbers), this paper addresses a different but equally critical threat: dataset-level property inference.

When Large Language Models (LLMs) are fine-tuned on domain-specific datasets (healthcare, finance, law), they may unintentionally memorize and reveal aggregate statistical properties of that data. For example, a medical LLM might reveal the prevalence of a sensitive disease (e.g., HIV) or the demographic breakdown (e.g., gender ratio) of the patients in its training set. Such leaks can expose strategic business intelligence or sensitive community demographics, even if no single individual's identity is compromised.

2. Methodology

The authors introduce PropInfer, a benchmark designed to evaluate property inference in LLMs across two distinct fine-tuning paradigms:

Q&A Mode (Supervised Fine-Tuning - SFT): The model learns to generate a response ( $y$ ) conditioned on an instruction and input ( $x$ ). This emphasizes the "doctor's" response.
Chat-Completion Mode (Causal Language Modeling - CLM-FT): The model learns the joint distribution of the entire sequence (instruction + input + output), potentially memorizing both the "patient's" input and the "doctor's" response.

To exploit these vulnerabilities, the paper proposes two novel attack vectors:

Black-Box Generation-Based Attack:
- Mechanism: The adversary uses carefully designed prompts to trigger the model to generate text.
- Process: The adversary generates multiple samples, uses a labeler (like GPT-4o) to determine if each sample contains the target property (e.g., "Is this patient female?"), and aggregates these labels to estimate the property ratio in the training set.
Grey-Box Shadow-Model Attack (Word-Frequency):
- Mechanism: The adversary leverages an auxiliary dataset to train "shadow models" with known property ratios.
- Process: Instead of using traditional features like loss or perplexity, the authors use word frequency. They identify a list of "keyword" signals that correlate strongly with the target property. A meta-attack model (XGBoost) is trained to map the word frequencies of a model's output to the underlying property ratio.

3. Key Contributions

PropInfer Benchmark: A standardized framework using the ChatDoctor dataset to study property inference across different fine-tuning modes and property types (demographics vs. medical diagnoses).
Tailored Attack Strategies: The development of two specific attacks (Generation-based and Word-frequency shadow models) that account for the unique linguistic and autoregressive nature of LLMs.
Empirical Analysis of Fine-Tuning Impact: A systematic study showing how the choice of fine-tuning (SFT vs. CLM-FT) changes the effectiveness of different attack types.

4. Key Results

The empirical evaluation across Llama and Pythia models yielded several critical findings:

Mode-Dependent Success:
- The Word-frequency attack is highly effective in Q&A Mode, particularly when the property is embedded in the user's input (e.g., gender cues in a patient's description).
- The Generation-based attack is highly effective in Chat-Completion Mode and when properties are present in both the question and the answer (e.g., medical diagnoses).
Superiority of Word Frequency: The word-frequency signal proved to be a more robust feature for property inference than traditional metrics like perplexity.
Pre-training Bias: The effectiveness of attacks can be influenced by the model's pre-trained knowledge. For sensitive topics like "childbirth," pre-training safety filters may suppress certain generations, making the attack harder.
Efficiency: The generation-based attack converges quickly, requiring as few as 500 samples to achieve low error rates.

5. Significance

This work identifies a previously unrecognized vulnerability in LLMs. It demonstrates that even if an LLM is "safe" regarding individual data extraction, it can still act as a statistical mirror of its training data, leaking sensitive aggregate information.

The paper highlights a tension between model utility and data confidentiality, suggesting that current fine-tuning practices are insufficient to protect dataset-level privacy. This provides a foundation for future research into robust defense mechanisms, such as specialized sampling methods or data subsampling, to mitigate the risk of property leakage in real-world deployments.