🤖 machine learning

Evaluating Differential Privacy Against Membership Inference in Federated Learning: Insights from the NIST Genomics Red Team Challenge

This paper empirically demonstrates that a novel stacking-based membership inference attack, which ensembles multiple black-box estimators, effectively compromises federated learning models even under high-privacy differential privacy settings ( $\epsilon=10$ ) and significantly outperforms traditional baselines in low-privacy scenarios ( $\epsilon=200$ ) where single-signal attacks fail.

Original authors: Gustavo de Carvalho Bertoli

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gustavo de Carvalho Bertoli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Secret Recipe Contest

Imagine a group of chefs (the Federated Learning clients) who all have secret family recipes. They want to create a "Super Chef" model that knows how to cook everything perfectly, but they don't want to share their actual recipe books with anyone because those recipes are their livelihood.

Instead of sending their books, they send updates (like "add a pinch more salt" or "cook at a lower temperature") to a central judge (the Server). The judge mixes these updates to create the Super Chef. This is Federated Learning (FL). It's supposed to be safe because no one ever sees the raw recipes.

The Problem: Even though the recipes are hidden, the Super Chef might accidentally "memorize" specific details about a specific chef's ingredients. A sneaky hacker (the Adversary) could ask the Super Chef questions and, based on how confident the Chef sounds, guess: "Did Chef A use this specific rare spice in their training?" If the answer is yes, the hacker knows that Chef A's secret data was used. This is called a Membership Inference Attack (MIA).

The Defense: The "Noise" Machine

To stop the hacker, the chefs decide to use Differential Privacy (DP). Think of this as a static-filled radio.

No Privacy: The radio is crystal clear. The hacker can hear every detail and guess perfectly.
Low Privacy: The radio has a little static. It's harder to hear, but a sharp-eared hacker might still catch a word.
High Privacy: The radio is full of loud static. The hacker can't hear anything at all.

The paper asks: Does this static actually stop the hacker, or is the hacker smart enough to tune it out?

The Hacker's New Trick: The "Super Detective" Team

Usually, hackers use a simple trick: they listen to the model's confidence. If the model is super confident, it probably saw that data before. But with the "static" (DP), the model sounds confused even when it knows the answer, making simple tricks fail.

The author of this paper built a Super Detective Team (an Ensemble Attack). Instead of one detective, they used seven different types of detectives (different AI algorithms like Random Forests, Neural Networks, etc.).

Each detective looks at the model from a slightly different angle.
One looks at how confident the model is.
Another looks at how much the model "stumbled" (loss) on the answer.
A third looks at the pattern of mistakes.

Then, they have a Team Leader (a Meta-Classifier) who listens to all seven detectives and makes the final call. Even if the static makes one detective confused, the others might still hear a clue. The Team Leader combines all these weak clues to make a strong guess.

The Experiment: The NIST Challenge

The author tested this Super Detective Team in a real-world contest organized by NIST (a US government standards agency) using soybean DNA data.

Scenario 1: No Static (No Privacy). The model was crystal clear. The Super Detective Team crushed the competition, guessing correctly about 53% of the time (which is huge in this game; random guessing is only 25%).
Scenario 2: Light Static (Low Privacy, $\epsilon=200$ ). The model had some noise. The simple hackers (using just one detective) failed completely. But the Super Detective Team still managed to guess correctly 38% of the time. They found "residual leakage"—tiny cracks in the armor that the single detectives missed.
Scenario 3: Heavy Static (High Privacy, $\epsilon=10$ ). The model was very noisy. The Super Detective Team's success dropped to 25% (basically random guessing). The static finally worked.

The Catch: The "Utility" Trade-off

Here is the twist. To make the static loud enough to stop the hacker (High Privacy), the chefs had to add so much noise that the Super Chef stopped learning how to cook.

In the High Privacy setting, the model's accuracy dropped so low it was useless (it couldn't even predict soybean colors better than a coin flip).
In the Low Privacy setting, the model was still a great chef, but the Super Detective Team could still sneak a peek at the recipes.

The Takeaway

Simple defenses aren't enough: If you just add a little bit of privacy noise, a smart, multi-tool hacker can still find your data.
Strong privacy kills utility: To truly stop these smart hackers, you need so much noise that the AI stops being useful.
The Future: We need better ways to protect data that don't break the AI. We can't just rely on "static" alone; we might need to lock the door (secure aggregation) or change the rules entirely.

In short: The paper shows that while Federated Learning is a great idea, it's not a magic shield. A clever hacker with a "team" of tools can still peek through the cracks, and fixing those cracks completely might break the whole system.

1. Problem Statement

The paper addresses the vulnerability of Federated Learning (FL) systems to Membership Inference Attacks (MIAs). While FL prevents raw data exchange, the resulting global or local models can still leak information about whether specific data records were part of the training set. This is particularly critical in sensitive domains like genomics.

The study focuses on the 2025 NIST Genomics Privacy-Preserving Federated Learning (PPFL) Red Teaming Event, which provided a controlled environment to evaluate the effectiveness of Differential Privacy (DP) as a defense against MIAs.

Threat Model: An untrusted central server (aggregator) that colludes with one client. The adversary has black-box access to client models (can query inputs and observe outputs) and access to auxiliary datasets (relevant and external records) but no access to private training gradients or internal parameters.
Dataset: High-dimensional genomic data (soybean seed coat color prediction) with 125,767 gene variants but very few samples per client (max 95). This $n \ll p$ (low sample, high dimension) regime induces severe overfitting, amplifying MIA signals.
Privacy Configurations: The study evaluates three tiers:
1. No DP: Standard CNN ( $\epsilon = \infty$ ).
2. Low Privacy: DP-SGD with $\epsilon = 200$ .
3. High Privacy: DP-SGD with $\epsilon = 10$ .

2. Methodology: Stacking-Based Ensemble Attack

The authors propose a novel black-box, stacking-based MIA designed to overcome the limitations of traditional shadow-model attacks (which are infeasible here due to the lack of private training data and small auxiliary pools).

A. Base Estimators (Signal Extraction)

Instead of relying on a single signal (e.g., just loss or just confidence), the attack employs seven diverse black-box estimators trained on the target model's outputs:

Neural Network (NN)
Random Forest (RF)
Decision Tree (DT)
Gradient Boosting (GB)
K-Nearest Neighbors (KNN)
Support Vector Machine (SVM)
Logistic Regression (LR)

These models are trained on noisy proxy labels: "Relevant" records (a mix of members and non-members) are treated as potential members, while "External" records (guaranteed non-members) are treated as non-members.

B. Meta-Feature Construction

For every target record, the system extracts an 8-dimensional feature vector:

7 continuous membership probabilities from the base estimators.
1 cross-entropy loss value ( $L_{CE}$ ) from the target model.

C. Meta-Classifier and Domain Adaptation

Meta-Model: An XGBoost classifier is trained to aggregate these heterogeneous signals.
Training Strategy: The meta-model is trained on Client 4, where ground-truth membership labels are fully known (due to the collusion assumption).
Domain Adaptation: To apply the model to Clients 1, 2, and 3 (where ground truth is unknown), the authors use a partial-fit strategy. The XGBoost model is fine-tuned using the "External" records (known non-members) of the target client to adjust for distribution shifts.
Decision Rule: A record is assigned to a specific client only if:
1. Its predicted probability exceeds a client-specific threshold ( $P_{thr}$ , the 55th percentile).
2. Its score is significantly higher (by a factor of $\lambda=1.5$ ) than the average score across all other clients.

3. Key Contributions

Shadow-Model-Free Attack: The paper introduces an attack strategy that replaces the canonical requirement for shadow models (trained on data from the same distribution as the target) with auxiliary-record-driven meta-learning. This makes the attack viable in realistic FL threat models where private data is inaccessible.
Empirical Characterization of DP: It provides a rigorous, third-party evaluation of how multi-signal ensemble attacks degrade across calibrated DP tiers ( $\epsilon = \infty, 200, 10$ ), revealing that residual leakage persists even at moderate privacy budgets where single-signal attacks fail.
Reproducible Benchmark: The work establishes a reproducible baseline for MIA research on the publicly available NIST Genomics dataset, utilizing an independent third-party evaluation platform.

4. Key Results

A. Attack Accuracy

The proposed ensemble method achieved the highest accuracy in the challenge for the No DP and Low Privacy tiers:

No DP ( $\epsilon = \infty$ ): 53.42% accuracy (vs. 30.14% for the next best baseline). This is statistically significant ( $p < 0.01$ ).
Low Privacy ( $\epsilon = 200$ ): 38.36% accuracy. Crucially, this outperformed the single-signal LiRA baseline (which collapsed to near-random performance), proving that ensemble methods can extract residual signals even when noise is present.
High Privacy ( $\epsilon = 10$ ): 24.66% accuracy. Performance converged with other baselines and approached the random assignment floor (~26%), indicating that strong DP effectively neutralizes the attack.

B. True Positive Rate (TPR) at Low False Positive Rates (FPR)

Evaluating TPR at strict FPR thresholds ( $\le 1\%$ ) on Client 4 revealed:

No DP: LiRA (single-signal) achieved 92.31% TPR, while the ensemble achieved 46.15%.
Low DP ( $\epsilon = 200$ ): LiRA dropped to 0.00% TPR. The ensemble maintained a 30.77% TPR.
Significance: This demonstrates that while DP noise destroys simple loss-based signals, the diversity of the ensemble (combining tree-based, distance-based, and parametric estimators) allows the meta-classifier to recover partial membership signals that single-metric attacks miss.

C. Privacy-Utility Trade-off

The paper simulates the FL training process to quantify the cost of privacy:

$\epsilon = 200$ : Preserves utility (83–85% accuracy) but leaves the model vulnerable to multi-signal attacks.
$\epsilon = 10$ : Effectively eliminates the attack but destroys model utility (accuracy drops to 16–47%, failing to converge).
Conclusion: In this high-dimensional genomic setting, there is a "gap" where low DP preserves utility but fails to protect privacy, and high DP protects privacy but renders the model useless.

5. Significance and Implications

Limitations of Standard DP Audits: The study argues that evaluating defenses against simple, single-signal attacks (like basic loss thresholding) is insufficient. Advanced, multi-signal ensemble attacks can bypass moderate DP protections ( $\epsilon=200$ ).
Practical Deployment Risks: For sensitive applications like genomics, relying solely on DP with a "low" budget ( $\epsilon=200$ ) may provide a false sense of security. Conversely, the "high" budget ( $\epsilon=10$ ) required for security is currently impractical for this specific task due to utility collapse.
Future Directions: The authors suggest that DP alone may be insufficient against sophisticated adversaries. Future defenses should explore complementary mechanisms such as secure aggregation or output perturbation to close the gap between utility and privacy without relying solely on extreme noise injection.

In summary, the paper demonstrates that while Differential Privacy is a necessary component of secure FL, its effectiveness is highly dependent on the sophistication of the adversary. Multi-signal ensemble attacks can exploit residual leakage in low-privacy settings, necessitating a re-evaluation of privacy-utility trade-offs in high-dimensional genomic federated learning.