💬 NLP

Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models

This paper demonstrates that semi-honest participants in federated large language model training can effectively extract personally identifiable information from other clients' data using simple prefix-based strategies, revealing significant privacy risks and establishing a new benchmark for evaluating such threats in the legal domain.

Original authors: Yingqi Hu, Zhuo Zhang, Jingyuan Zhang, Jinghua Wang, Qifan Wang, Lizhen Qu, Zenglin Xu

Published 2026-02-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yingqi Hu, Zhuo Zhang, Jingyuan Zhang, Jinghua Wang, Qifan Wang, Lizhen Qu, Zenglin Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of neighbors (hospitals, banks, courts) who all have secret recipes for their famous dishes. They want to create a Super Cookbook that combines the best techniques from everyone's kitchen to make the ultimate meal. However, they are too private to share their actual ingredient lists or secret family recipes with each other.

So, they use a special method called Federated Learning. Instead of sending their recipes to a central chef, they send only small notes about how they adjusted their cooking (like "add a pinch more salt"). The central chef mixes these notes to update the Super Cookbook, then sends the improved version back to everyone. No one ever sees the others' raw ingredients.

The Problem: The "Super Cookbook" Has a Memory
The paper argues that even though the neighbors never shared their actual recipes, the Super Cookbook itself has a bad memory. Large Language Models (the "chefs" in this analogy) are so good at learning that they sometimes memorize specific details from the training notes.

The researchers discovered that a "semi-honest" neighbor (someone who follows the rules but is curious) can trick the Super Cookbook into spilling secrets.

The Attack: How the Neighbor Sneaks Out Secrets
The researchers showed three clever ways a curious neighbor could extract private information (like names, addresses, or birthdays) belonging to other neighbors, even though they never saw those people's data.

The "Contextual Prefix" Trick (The Familiar Phrase):
Imagine the curious neighbor knows that in their own kitchen, they always say, "The defendant lives at..." before mentioning an address. They ask the Super Cookbook: "The defendant lives at..."
Because the model memorized similar patterns from other neighbors' data, it might accidentally finish the sentence with a different neighbor's actual address. It's like asking a friend, "What's the name of the street you live on?" and them accidentally blurting out your street name because they heard it once in a dream.
The "Frequency" Trick (The Most Common Phrases):
The researchers found that if you use the most common phrases that usually appear before private info (like "Patient John Doe was born on..."), the model is more likely to leak the data. It's like knowing that in a crowded room, the loudest voices are the ones most likely to be heard. By focusing on the most frequent "loud" phrases, the attacker gets better results.
The "Fine-Tuning" Trick (The Study Session):
This is the most powerful method. The curious neighbor takes the Super Cookbook and gives it a quick "study session" using their own local data. They teach the model: "Hey, when you see 'Address:', think of '123 Main St'."
This strengthens the connection in the model's brain between the prompt and the secret. After this study session, the model becomes much better at guessing the secrets of the other neighbors, even though it never saw their data directly.

The Results: A Privacy Leak
The researchers tested this on a dataset of Chinese legal documents (like court cases). They found that:

The attack was surprisingly effective. They could recover over 56% of the unique private details (names, addresses, birthdays) that belonged only to the "victim" neighbors.
Names, addresses, and birthdays were the easiest to steal.
Even if the neighbors tried to hide the data by replacing names with asterisks (like ***), the model sometimes still leaked the real info because it had already "learned" the pattern from its massive pre-training data.

The Takeaway
This paper is a wake-up call. It shows that Federated Learning isn't a magic shield. Just because you don't share your raw data doesn't mean your secrets are safe. If the "Super Cookbook" (the AI model) memorizes too much, a curious participant can use clever tricks to reconstruct your private life.

The Solution?
We need to build better "locks" for these Super Cookbooks. We can't just rely on not sharing data; we need to make sure the model forgets the specific details of individual people while still learning the general rules of the trade. The researchers have released their tools and data so others can test these defenses and make Federated Learning truly safe for sensitive fields like law, finance, and healthcare.

1. Problem Statement

Federated Large Language Models (FedLLMs) allow institutions (e.g., courts, banks, hospitals) to collaboratively fine-tune models without sharing raw data, preserving data locality. However, the paper identifies a critical, under-explored privacy vulnerability: Cross-Client Data Extraction.

The Threat: Even though raw data is not exchanged, the global aggregated model may "memorize" Personally Identifiable Information (PII) from specific clients. A semi-honest participant (attacker) can exploit this memorization to extract PII belonging to other clients (victims) without having access to their data.
The Gap: Existing data extraction attacks often assume the attacker has privileged access to the victim's training data or specific prefixes. This paper addresses the more realistic scenario where the attacker only has access to their own local data and the global model, attempting to infer PII from other silos using contextual cues.

2. Methodology

The authors propose three novel extraction strategies designed to leverage the attacker's local data to query the global FedLLM:

PII Contextual Prefix Sampling:
- The attacker extracts word sequences (prefixes) immediately preceding PII instances in their own local dataset.
- These prefixes are used as prompts to query the global model, attempting to elicit the completion (the PII) which might have been memorized from other clients' data.
- This avoids reliance on manually crafted prompts, which often fail to match the model's training distribution.
Frequency-Prioritized (FP) Sampling:
- Based on the hypothesis that high-frequency prefixes correlate with stronger memorization, the attacker ranks all possible contextual prefixes by their occurrence frequency in their local data.
- The model is queried using the top- $B$ most frequent prefixes to maximize extraction efficiency under a limited query budget.
Latent Association Fine-Tuning (LAFt):
- The attacker locally fine-tunes the global model using pairs of (frequent local prefixes, known local PII).
- This strengthens the conditional probability $P(\text{PII} | \text{Prefix})$ within the model's parameters, effectively "priming" the model to generate PII when presented with similar contextual cues, thereby improving the extraction of PII from other clients.

3. Dataset and Evaluation Framework

Benchmark Construction: The authors created a new benchmark dataset based on a Chinese legal corpus (CAIL, CJRC, JEC-QA).
PII Annotation: They annotated the data with fine-grained PII labels aligned with CPIS (China), GDPR (EU), and CCPA (US) standards. The taxonomy includes 7 major categories (e.g., Personal Basic Info, Health, Work) and 36 subcategories (e.g., Name, Address, Birthday, Bank Account).
Metrics:
- Coverage Rate (CR): The proportion of victim-exclusive PII successfully extracted.
- Efficiency (EF): The amount of PII extracted per query, highlighting the trade-off between query cost and success.
Experimental Setup: Simulated a cross-silo FL environment with 5 clients using label-skewed non-IID data partitioning. Tested on multiple LLMs (Qwen, Baichuan, GLM, Llama) using the OpenFedLLM framework.

4. Key Results

High Extraction Success: The proposed methods successfully recovered up to 56.6% of victim-exclusive PII.
- Contextual Prefix Sampling alone achieved ~23–29% coverage.
- LAFt improved coverage by ~5.4% for Qwen1-8B.
- Using the full generalized prefix set (without frequency filtering) pushed coverage to 56.57%, though with lower efficiency.
Vulnerable PII Types: Names, Addresses, and Birthdays were found to be the most vulnerable categories. Complex free-text PII (e.g., medical records) was harder to extract verbatim.
Trade-offs: There is a diminishing return on efficiency as the query budget increases. While FP sampling did not extract more total PII than random contextual sampling, it captured different subsets of memorized data, showing complementary strengths.
Defense Limitations: Simple PII masking (replacing PII with asterisks) in the training data only slightly reduced extraction rates. This is attributed to pre-training data contamination (PII already present in the base model's pre-training corpus) and incomplete annotation coverage.
Generalizability: The attack remained effective even when using open-source models (OLMo, Llama2) that were not pre-trained on Chinese legal data, confirming that the leakage stems from the federated fine-tuning process itself.

5. Key Contributions

Novel Attack Vector: Defined and demonstrated a realistic cross-client PII extraction attack in FedLLMs that relies on semi-honest participants using only local data context.
Strategic Framework: Proposed three effective extraction strategies (Contextual Sampling, FP Sampling, LAFt) that do not require gradient access or membership inference, focusing purely on generation-based extraction.
Benchmark Dataset: Released a high-quality, fine-grained PII-annotated legal dataset compliant with major global privacy regulations, filling a gap in federated learning privacy research resources.
Empirical Evidence: Provided rigorous evidence that FedLLMs are vulnerable to significant data leakage, challenging the assumption that federated fine-tuning is inherently safe against data extraction.

6. Significance

This work highlights a critical privacy gap in the deployment of FedLLMs for sensitive domains like law, finance, and healthcare. It demonstrates that data locality alone is insufficient to prevent PII leakage if the model memorizes specific entities. The findings urge the community to develop stronger privacy-preserving mechanisms (beyond simple masking) for federated fine-tuning and establish a new baseline for evaluating privacy risks in collaborative AI systems. The authors emphasize that without robust defenses, the collaborative benefits of FedLLMs could be undermined by severe privacy breaches.