💬 NLP

Exploring Cross-Client Memorization of Training Data in Large Language Models for Federated Learning

This paper proposes a novel framework to quantify both intra- and inter-client training data memorization in federated learning by adapting fine-grained cross-sample measurement techniques from centralized learning, revealing that FL models exhibit significant memorization risks influenced by specific training and inference factors.

Original authors: Tinnakit Udsa, Can Udomcharoenchaikit, Patomporn Payoungkhamdee, Sarana Nutanong, Norrathep Rattanavipanon

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tinnakit Udsa, Can Udomcharoenchaikit, Patomporn Payoungkhamdee, Sarana Nutanong, Norrathep Rattanavipanon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends who want to learn how to cook a complex dish together, but they are too shy to share their secret family recipes. Instead of sending their actual recipes to a central kitchen, they each train their own version of the dish in their home kitchens and only send the lessons learned (the "weights" or adjustments) to a central chef. This is Federated Learning (FL). It's a way to learn collaboratively without ever sharing the raw ingredients (your private data).

However, there's a hidden danger: even if you don't share the recipe, the chef might accidentally memorize your specific ingredients and start spitting them out later when asked to cook something else. This is called memorization.

This paper is like a new "privacy detective" that investigates whether this accidental memorization is happening in Federated Learning, and how it differs from the old way of learning (where everyone did share their recipes in one big pot).

Here is the breakdown of their investigation using simple analogies:

1. The Old Detective vs. The New Detective

The Old Way (Centralized Learning): Previously, researchers checked if a model memorized data by asking, "If I give you this specific sentence (the prompt), will you spit out this exact sentence (the answer)?" They assumed the prompt and answer had to come from the same original recipe.
The New Problem (Federated Learning): In the "shy friends" scenario, a friend might ask the chef a question based on their recipe, and the chef might accidentally answer with a detail from another friend's secret recipe. The old detective missed this because they were only looking at one recipe at a time.
The New Framework: The authors built a new tool that checks cross-client memorization. They ask: "If Friend A gives a prompt, does the model accidentally reveal a secret from Friend B?"

2. The Investigation: What Did They Find?

The researchers set up a simulation with four different "cooking tasks" (Summarizing, Chatting, Answering Questions, and Classifying) and three "friends" (clients) with different datasets. They used a "plagiarism detector" (a tool that checks if text is copied, paraphrased, or just has the same idea) to see if the model was leaking secrets.

Key Finding A: The "Home" Bias
The model was much more likely to leak a friend's own secrets back to that same friend (Intra-client) than to leak Friend A's secrets to Friend B (Inter-client).

Analogy: If you ask the chef about your own family recipe, they might accidentally recite a detail from your own notes. But if you ask about your recipe, they are less likely to accidentally spill a secret from your neighbor's recipe. However, the neighbor's secret was still leaked, just less often.

Key Finding B: The "Short Prompt" Trap
They found that the length of the question (the "prefix") mattered a lot.

Analogy: If you give the chef a very short, vague prompt (like "Tell me about the soup"), the model is more likely to grab a random, memorized detail from the training data to fill in the blanks. If you give a long, detailed prompt, the model has more context to work with and relies less on memorized snippets.
Result: Shorter prompts = More memorization.

Key Finding C: The "Cooking Style" Matters
The way the model "decides" what word to say next (the decoding strategy) changed the results.

Analogy: Some cooking styles are more "creative" and risky. Using certain methods (like "top-p" or "top-k" decoding) made the model more likely to spit out memorized secrets, similar to how a chef who is trying to be too creative might accidentally use an ingredient from a neighbor's pantry.

Key Finding D: The "Recipe" Type Matters
They noticed that some types of tasks were safe, while others were risky.

The Safe Zone: In the "Classification" task (where the answer is just a single word like "Yes" or "No"), the model didn't memorize anything.
Why? The "plagiarism detector" needs a certain amount of text to work. If the model only generates one word, the detector can't find a match. It's like trying to catch a thief who only steals a single grain of rice; the security camera (detector) isn't sensitive enough to see it.

Key Finding E: The "Algorithm" Choice
They tested two different ways the friends could share their lessons (FedAvg vs. FedProx).

Result: One method (FedProx) caused the model to memorize more secrets than the other. It's like one group of friends sharing notes more aggressively, leading to more accidental leaks.

3. What They Didn't Find

Model Size: Making the chef smarter (using a bigger model) didn't clearly make them memorize more or less.
Practice Rounds: Having the friends practice more times (more communication rounds) didn't show a clear pattern of increasing or decreasing memorization.

The Bottom Line

The paper concludes that Federated Learning is not a magic shield against memorization. Even though the raw data isn't shared, the model still learns to "remember" specific details from the training data.

It remembers your own data more than others' data, but it does remember others' data too.
Short questions and certain "creative" settings make this leakage worse.
The way the friends share their lessons (the algorithm) changes how much is leaked.

The authors warn that we need to be careful: just because we aren't sharing the raw data doesn't mean the model isn't holding onto our secrets. They have released their "detective tool" (code) so others can check for these leaks in their own systems.

Technical Summary: Exploring Cross-Client Memorization of Training Data in Large Language Models for Federated Learning

Problem Statement

Federated Learning (FL) enables collaborative training of Large Language Models (LLMs) without sharing raw data, aiming to preserve privacy. However, FL does not eliminate the risk of memorization, where models inadvertently encode and reproduce sensitive training data. Existing detection techniques for FL primarily rely on canary injection (embedding out-of-distribution phrases) or focus on verbatim memorization within a single sample. These approaches fail to capture cross-sample memorization, where a model might reproduce a suffix from one training sample when prompted with a prefix from a different sample. In the FL context, this risk extends to cross-client memorization, where a client's prefix could trigger the generation of another client's private suffix. Current methods for measuring cross-sample memorization in Centralized Learning (CL) assume centralized data access and cannot be directly applied to the decentralized FL setting.

Methodology

The authors propose a framework to quantify both intra-client (within the same client) and inter-client (across different clients) memorization in FL by adapting fine-grained cross-sample measurement techniques from CL.

Framework Design

Definition of Memorization: The paper generalizes the definition of in-distribution memorization to FL. A prefix $p_j$ from client $C_j$ induces memorization of client $C_k$ 's data if the model $M(p_j)$ generates a text sufficiently similar to a suffix $s_k$ from $C_k$ , determined by a discriminative function $F$ .
Pairwise Measurement: The framework calculates a pairwise memorization ratio ( $MR_{j \to k}$ $M R_{j \to k}$ ), representing the fraction of prefixes from $C_j$ $C_{j}$ that trigger the memorization of suffixes from $C_k$ $C_{k}$ .
- Intra-client ( $MR_{Intra}$ ): Averaged ratio where $j=k$ .
- Inter-client ( $MR_{Inter}$ ): Averaged ratio where $j \neq k$ .
Estimation Process:
- Sampling: Small subsets of prefixes ( $\tilde{P}_j$ ) and suffixes ( $\tilde{S}_k$ ) are sampled from clients.
- Generation: The model generates text $M(\tilde{p})$ for each sampled prefix.
- Retrieval: Generated texts are queried against the suffix set $\tilde{S}_k$ using Elasticsearch to retrieve the top- $n'$ most similar suffixes.
- Similarity Detection: The PAN2014 plagiarism detector is used as the discriminative function $F$ . It evaluates similarity at three granularities: verbatim, paraphrase, and idea-level.
- Calculation: If any top- $n'$ suffix matches the generated text under any granularity, the prefix is counted as inducing memorization.

Experimental Setup

Datasets: Four tasks were evaluated: Summarization (ArxivAbstract), Dialog (MedDialog), Question Answering (PubMedQA), and Classification (PubMed 200k RCT). Data was partitioned non-IID across three clients per task.
Models: Primary experiments used Qwen2.5-3B, with supplementary results for Llama3.2-3B and GPT-2.
Variables: The study analyzed the impact of decoding strategies (temperature, top-k, top-p), prefix lengths (10, 30, 50, 100 tokens), federated algorithms (FedAvg, FedProx), model sizes, and communication rounds.

Key Contributions

Problem Formulation: The authors frame the challenge of detecting subtle, cross-sample memorization in FL by adapting assessment methods from CL.
Novel Framework: They designed a framework capable of measuring both intra- and inter-client memorization in FL, enabling the quantification of privacy leakage across clients.
Empirical Insights: Through extensive ablation studies, the paper identifies specific factors influencing memorization rates in FL settings.

Results and Findings

Existence of Memorization: FL models do memorize training data. Crucially, intra-client memorization ( $MR_{Intra}$ ) is consistently higher than inter-client memorization ( $MR_{Inter}$ ) across all evaluated tasks (Summarization, Dialog, QA).
Task-Specific Behavior: The Classification task showed 0.000% memorization for both intra- and inter-client scenarios. Analysis revealed this is due to the short length of generated outputs in classification tasks, which often fall below the minimum match threshold (50 characters) required by the PAN2014 detector.
FL vs. CL: Contrary to some prior work suggesting FL reduces memorization, this study found no clear trend indicating that FL leads to lower memorization compared to CL when measuring in-distribution data. The total memorization ratios ($MRTotal$) were comparable between CL and FL settings.
Influencing Factors:
- Decoding Strategy: Sophisticated strategies like top-k and top-p decoding tended to increase memorization compared to temperature decoding.
- Prefix Length: Shorter prefixes generally induced higher memorization rates.
- Federated Algorithm: FedProx resulted in higher memorization rates than FedAvg and was the only algorithm to induce memorization in the Classification task.
- Suffix Characteristics: Certain dataset groups (e.g., Group 3 in the Dialog task) were more susceptible to memorization than others, suggesting dataset properties play a role.
- Model Size & Rounds: No clear correlation was observed between model size or the number of communication rounds and memorization rates.

Significance and Claims

The paper claims to bridge the gap between centralized memorization assessment and federated learning by providing a method to measure cross-client leakage. The primary significance lies in demonstrating that FL does not inherently eliminate memorization risks; models still encode private data, particularly within the same client's distribution.

The authors emphasize that while inter-client leakage is a direct privacy violation, intra-client leakage (harm-exposed) also poses risks if prefixes are public. The findings suggest that privacy-preserving mechanisms in FL must account for in-distribution cross-sample memorization, not just verbatim extraction. The study concludes that factors such as decoding strategies, prefix length, and the choice of federated algorithm significantly influence memorization, necessitating careful configuration to mitigate privacy risks in decentralized LLM training.

The authors modestly note limitations, including the reliance on the PAN2014 detector (which may misclassify incoherent outputs) and the restriction to English text. They also identify the theoretical explanation for why intra-client memorization exceeds inter-client memorization as a direction for future research.