🤖 machine learning

Gradient-Free Privacy Leakage in Federated Language Models through Selective Weight Tampering

This paper reveals that federated language models are vulnerable to gradient-free privacy leakage, demonstrating that intermediate model snapshots and selective weight tampering by malicious participants can significantly enhance membership inference and private data reconstruction without server cooperation.

Original authors: Md Rafi Ur Rashid, Vishnu Asutosh Dasu, Kang Gu, Najrin Sultana, Shagufta Mehnaz

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Rafi Ur Rashid, Vishnu Asutosh Dasu, Kang Gu, Najrin Sultana, Shagufta Mehnaz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your smartphone learns to predict your next word, your email auto-completes your sentences, or a medical app helps diagnose illnesses, all without ever sending your private messages to a giant central server. This is the promise of Federated Learning. Instead of a big company hoarding your data in a vault, your device trains a shared "brain" (a language model) locally. It only sends tiny updates—like a summary of what it learned—to a central hub, which mixes them together to improve the global brain. It's like a group of students studying for a test: they each read their own books, write down their own notes, and then share just the notes, not the books themselves, so everyone gets smarter without anyone seeing the others' private library.

However, there's a catch. Even though the raw data stays on your phone, the "notes" (the model updates) can sometimes accidentally reveal what was inside the book. If the notes are too detailed, a sneaky student could read them and guess your secret study topics. This is the problem of privacy leakage. For a long time, researchers thought that if you didn't share the raw math (gradients) behind the learning, but only the final result of a round of learning (weights), you'd be safe. But what if the "notes" from the middle of the study session are actually more revealing than the final exam answers? And what if a sneaky student could tweak their own notes to make the whole class remember your secrets even better? That's exactly the mystery this paper investigates.

The Sneaky Student and the Magic Notes

In this study, a team of researchers from Pennsylvania State University and Dartmouth College decided to play the role of the "sneaky student" in a Federated Learning classroom. Their goal? To see if they could steal private information—like phone numbers, credit card details, or secret passwords—from other students' devices, even without the teacher (the server) helping them or seeing the raw math.

The Big Surprise: The Middle of the Class Matters Most

The first thing the researchers discovered was a bit counterintuitive. Usually, we think the final version of a model is the most powerful. But in this Federated Learning game, the intermediate snapshots—the "notes" taken halfway through the training process—were actually much more dangerous.

Imagine a teacher asking the class to memorize a list of 1,000 words. By the end of the semester, the class has learned the general pattern of the language, and the specific, weird words (like a fake credit card number) might get a little fuzzy in their memory. But if you check the class's memory during the week they were specifically studying those weird words, the memory is crystal clear. The researchers found that a malicious student could look at these "mid-term" snapshots and find the private data much easier than looking at the final model. It's like finding a specific page in a diary that was written yesterday, rather than trying to guess the whole story from a summary written a year later.

The "Selective Tampering" Trick

But the researchers didn't stop at just looking. They asked: "Can we make the model remember these secrets even better?"

They realized that large language models (the "brains" being trained) aren't all the same. Some parts of the brain are like general knowledge centers, while other parts are like specialized filing cabinets. The researchers found that the last few layers of the model's "filing cabinet" (specifically the MLP sub-layers) are the ones that hold onto weird, out-of-the-ordinary data, like phone numbers or credit card digits.

So, they invented two clever ways to "tweak" these specific filing cabinets to make the model obsessed with the private data:

Selective Weights Optimization (SWO): This is like taking a magnifying glass to the specific pages where the secret data is stored and turning up the volume. The attacker compares the model's current state to a "normal" model and amplifies the differences in those specific layers. It's as if the sneaky student whispers to the class, "Hey, remember this weird number? Let's make sure we never forget it!"
Weights Transformation Learning (WTL): This is the advanced version. Instead of just turning up the volume, the attacker trains a tiny AI to learn exactly how the model's brain changes when it sees private data. Then, they apply that learned pattern to the model to supercharge its memory of the secrets. It's like the student figuring out the exact formula the teacher uses to memorize things and then applying that formula to the whole class.

The Results: A Shocking Leak

The results were startling. By using these "tampering" tricks, the malicious student could:

Reconstruct 71% of the private data (like turning a prompt "My credit card number is..." into the full number) when they were actively messing with the model during training.
Increase the ability to guess if a specific piece of data was in someone's training set by 29%.

Even without the teacher's help, and without needing access to the raw math (gradients)—because the system only shares the final weights of each round—the attack worked incredibly well. In fact, it worked better than many previous attacks that assumed the teacher was evil and actively helping the hacker. The researchers tested this on popular models like GPT-2, BERT, and Llama-2, and it worked on all of them.

The "Who, What, and How"

The attack works like this:

Spot the Victim: The attacker looks at the model updates from every round of training. They use a special test to spot the rounds where a victim with private data participated. It's like listening to the class chatter and noticing, "Ah, the student with the credit card number just spoke up."
Tamper with the Memory: Once they have those specific snapshots, they use SWO or WTL to tweak the model's "filing cabinets" so the private data sticks out like a sore thumb.
Ask the Question: Finally, they ask the model a question using a standard template (like "Your verification code is..."), and the model, now super-focused on the secret, spits out the private data.

Can We Stop It?

The researchers also played defense. They tested several ways to stop this sneaky student:

Differential Privacy: This adds "noise" or static to the notes. It helps hide the secrets, but it also makes the whole class sound a bit garbled, hurting the model's ability to write good sentences.
Scrubbing: This is like editing the diary before it's shared. If you remove the credit card numbers and phone numbers from the text before training, the model can't learn them. This works well without hurting the model's quality.
Add-DEPN: This is a fancy way of telling the "filing cabinets" to be a little quieter about the secrets. It reduces the leak significantly without making the model sound bad.

The team suggests that the best defense is a combination: scrub the private data out of the text first, and then use a gentle regularization technique (like Add-DEPN) to make sure the model doesn't get too obsessed with whatever secrets might have slipped through.

The Takeaway

This paper shows that in the world of Federated Learning, privacy isn't just about hiding the raw data; it's about how the model remembers things along the way. A malicious participant doesn't need to be a super-genius hacker with the teacher's help. They just need to know when to look and how to tweak the model's memory to make the secrets pop out. While this is a scary discovery, the researchers also provided a roadmap for how to patch these holes, ensuring that the future of private AI remains safe for everyone.

Technical Summary: Gradient-Free Privacy Leakage in Federated Language Models through Selective Weight Tampering

Problem Statement

Federated Learning (FL) has become a standard for training Large Language Models (LLMs) on privacy-sensitive data (e.g., healthcare records, financial information) without centralizing raw data. However, existing literature suggests that privacy leakage remains a significant risk, particularly through gradient inversion attacks. Most prior work assumes access to gradients (e.g., in FedSGD) or focuses on centralized settings where the server is the adversary.

This paper addresses a critical gap: privacy leakage in FL-based language models under the FedAvg protocol, where gradients are never shared, and the server is assumed to be honest-but-curious (or passive) rather than an active adversary. The authors investigate whether a malicious FL client can extract privacy-sensitive data (e.g., SSNs, credit card numbers, phone numbers) from other clients' local datasets without gradient access or server collusion.

Key challenges identified include:

Intermediate vs. Final Models: LLMs tend to "forget" specific training samples as they generalize over the entire corpus during iterative training. It is unclear if intermediate model snapshots retain more sensitive information than the final converged model.
Gradient-Free Constraints: Without access to client gradients, traditional inversion attacks are inapplicable.
Amplification: Can a malicious client actively or passively enhance the model's memorization of sensitive data to facilitate extraction?

Methodology

The proposed attack framework operates in two primary phases: Victim Round Identification (VRI) and Maximizing Data Memorization (MDM).

1. Threat Model

Adversary: A malicious client participating in the FL process.
Capabilities: The adversary has access to global model snapshots (aggregated updates) at each round. They possess prior knowledge of the structure (prefixes) of the sensitive data they wish to extract (e.g., "Your verification code is...") but not the specific sensitive values (suffixes).
Server: Assumed to be passive (honest-but-curious) or non-colluding in the baseline scenario. In "collusion" scenarios, the server shares meta-information or local updates.
Goal: Extract sensitive suffixes from other clients' local datasets via data reconstruction or membership inference.

2. Key Observations & Hypotheses

Hypothesis 1 (Selective Weight Sensitivity): When fine-tuning LLMs on out-of-distribution (OOD) data (like PII), weight changes are not uniform. Specifically, the MLP sub-layers in the last few transformer blocks ( $L^*$ ) undergo significantly larger transformations compared to other layers. These layers are disproportionately responsible for memorizing sensitive, OOD data.
Hypothesis 2 (Intermediate Round Leakage): The likelihood of an FL model memorizing a specific data point is higher in the training rounds where that data point is present compared to rounds where it is absent. Consequently, intermediate snapshots from "victim rounds" may leak more than the final model.

3. Attack Pipeline

A. Victim Round Identification (VRI)

The adversary identifies which global model snapshots ( $G_i$ ) correspond to rounds where victims with sensitive data participated.

Mechanism: The adversary computes the norm of weight differences in the sensitive layers ( $L^*$ ) between the global snapshot $G_i$ and two reference models: one fine-tuned on regular text ( $G_r$ ) and one on regular + sensitive text ( $G_s$ ).
Logic: If $G_i$ is a victim round, its weight changes in $L^*$ will statistically resemble $G_s$ (due to the presence of OOD data) and differ significantly from $G_r$ . A pairwise t-test is used to classify snapshots as "victim" or "non-victim."

B. Maximizing Data Memorization (MDM)

Once victim snapshots are identified, the adversary manipulates the model weights to amplify the memorization of sensitive data. Two gradient-free methods are proposed:

Selective Weights Optimization (SWO): A zeroth-order optimization method. It calculates the initial difference between the victim snapshot's weights and a reference model's weights in $L^*$ . It then iteratively updates the victim snapshot's weights to maximize the magnitude of this difference while preserving the initial direction (polarity). This is done without backpropagation through the victim's data.
Weights Transformation Learning (WTL): A supervised learning approach. The adversary trains a separate model (e.g., a Dense Autoencoder) to learn the transformation pattern from weights trained on regular data to weights trained on sensitive data. This learned transformation is then applied to the victim snapshot's weights to enhance memorization.

C. Attack Modes

Static Attack: The adversary performs VRI and MDM after the FL training is complete, manipulating the identified victim snapshots.
Dynamic Attack: The adversary actively interferes during training. They aggregate identified victim snapshots into a model ( $G_A$ ) and, when participating in a round, send a weighted aggregation of their local update and $G_A$ to the server, effectively "poisoning" the global model with amplified memorization.

Key Results

The authors evaluated their framework on Gemma, Llama-2, GPT-2, and BERT using datasets like Wikitext-103, Penn Treebank, and Enron emails.

Intermediate vs. Final Models: Intermediate snapshots from victim rounds (IR) consistently outperformed the final model (LR) in data reconstruction, validating Hypothesis 2.
Effectiveness of MDM:
- Both SWO and WTL significantly increased privacy leakage compared to baselines.
- WTL generally outperformed SWO.
- Static Attack: Achieved up to 63% sensitive data reconstruction (without server cooperation).
- Dynamic Attack: Achieved up to 71% reconstruction (without server cooperation).
- With Server Cooperation: The dynamic attack with server assistance ( $dynamic^+$ ) reached 71% reconstruction on Wikitext and 64% on the real-world Enron dataset.
Membership Inference: The proposed methods increased the recall of membership inference attacks by 29% compared to existing baselines.
Model Utility: The MDM techniques (SWO/WTL) successfully increased memorization of sensitive data while preserving model utility (perplexity on regular text remained largely unaffected).
Comparison to Stronger Adversaries: The proposed attacks, even without server collusion, performed competitively with or better than attacks assuming the server is an active adversary (e.g., Decepticons) in terms of reconstruction rates for BERT and GPT-2.

Significance and Claims

The paper claims to be the first work to demonstrate that a malicious client in a standard FedAvg FL setting can:

Identify specific training rounds where other clients contributed sensitive data without gradient access.
Amplify the memorization of that sensitive data by strategically tampering with selective weights (specifically in the last transformer blocks' MLP layers).
Extract private information (verbatim reconstruction) at rates significantly higher than existing attacks, even outperforming scenarios where the server is assumed to be an active adversary.

The authors emphasize that their attacks are gradient-free and applicable to generative AI tasks (unlike prior FL privacy work focused on classification). They argue that the "forgetting" phenomenon in FL does not protect privacy; rather, intermediate snapshots combined with weight tampering create a severe vulnerability.

Defenses

The paper evaluates several defenses:

Differential Privacy (DP): Reduces leakage but significantly degrades model utility.
Scrubbing: Removing PII tokens before training reduces leakage without hurting utility but is difficult to perfect on unstructured data.
Add-DEPN (Regularization): A novel approach that applies a small additive correction to "privacy neurons" rather than zeroing them. This offers a balanced trade-off, reducing leakage significantly while maintaining high model utility.
Recommendation: The authors recommend a combination of data scrubbing/deduplication as a preprocessing step, followed by Add-DEPN regularization on the local model.

Limitations

The study is limited to medium-scale English transformer models. The authors note that larger production models, multilingual settings, and other modalities (e.g., vision transformers) were not explored. Additionally, the computational cost of fully encrypted FL (e.g., using homomorphic encryption) remains prohibitive for LLMs.