🤖 machine learning

Auditing Fairness-Privacy Trade-offs: Subpopulation-Level Effects of Fairness-Enhancing Algorithms

This paper presents the first comprehensive study demonstrating that fairness-enhancing algorithms have heterogeneous, subpopulation-specific effects on membership inference privacy risks, revealing that the interplay between fairness, privacy, and utility requires joint evaluation at the subgroup level rather than through aggregate metrics.

Original authors: Umid Suleymanov, Ilhama Novruzova, Khalid Mammadov, Natavan Hasanova, Murat Kantarcioglu

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Umid Suleymanov, Ilhama Novruzova, Khalid Mammadov, Natavan Hasanova, Murat Kantarcioglu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a giant, bustling library where every book represents a person's life story. In this library, there are two very important rules that the librarians (the computer programs) must follow. The first rule is Fairness: the librarians must treat every visitor equally, regardless of whether they are tall, short, wear glasses, or come from a specific neighborhood. They can't give better service to one group while ignoring another. The second rule is Privacy: the librarians must keep the visitors' secrets safe. They shouldn't be able to tell a stranger, "Hey, I know you were here yesterday!" just by looking at how they organized the books.

For a long time, scientists have been worried that these two rules might fight each other. They thought that if you tried to make the library super fair, you might accidentally make it easier for snoops to guess who visited. Or, if you tried to hide everyone's identity perfectly, the librarians might get so confused that they started treating people unfairly. But here's the twist: most scientists were only looking at the library as a whole. They were checking the average treatment of everyone, missing the fact that some specific groups of visitors might be getting a totally different, and much worse, deal than others. It's like saying the "average" temperature in a city is perfect, while ignoring that one neighborhood is freezing and another is baking.

This paper decides to zoom in and look at the library through a magnifying glass, checking each specific group of visitors one by one. The researchers wanted to see what happens when you use special "fairness tools" to fix the librarians' behavior. Do these tools accidentally make it easier for a snoop to guess which visitors are in the system? And if you try to hide everyone's identity using a "privacy shield" (a technique called Differential Privacy), does it help everyone equally, or does it accidentally hurt the smaller, less common groups even more?

The team set up a massive experiment using ten different real-world scenarios, ranging from predicting who gets a loan to figuring out who might re-offend in the legal system. They tested five different types of "fairness tools" (some that fix the data before the computer learns, some that change how the computer learns, and some that fix the answers after the computer is done). They also used three different types of "snoop attacks" to see how easy it was to guess if a person was in the training data.

Here is what they discovered, and it's a bit more complicated than a simple "good vs. bad" story. First, they found that fairness and privacy are not enemies. You don't have to choose between them; in fact, some fairness tools actually made it harder for snoopers to guess who was in the system, while others made it easier. It all depends on which tool you use and who is in the group. For example, one tool called "Reweighing" (which gives extra attention to smaller groups) often made the smaller groups safer from snoopers. But another tool, "Disparate Impact Remover," was a bit of a rollercoaster—it sometimes helped and sometimes hurt, depending on the specific dataset.

The most surprising finding came when they combined fairness tools with the "privacy shield" (Differential Privacy). The researchers found that while the privacy shield successfully stopped the snoopers, it did so by throwing a blanket of "noise" over the whole library. This noise was supposed to hide everyone's identity, but it turned out to be a heavy blanket that crushed the smaller, less common groups. For the big, popular groups, the librarians could still do their jobs well even with the noise. But for the tiny, rare groups, the noise was so loud that the librarians couldn't tell anything at all, and their accuracy dropped to near zero. It's like trying to hear a whisper in a quiet room; if you suddenly turn on a loud fan to hide the whisper, the big voices can still be heard, but the whispers are completely lost.

The paper also looked at a tool called "Calibrated Equalized Odds," which adjusts the final answers to be fair. This tool was great at hiding privacy risks (making the snoopers guess randomly), but it was a disaster for accuracy. It didn't just help the underrepresented groups; it shuffled the errors around in a chaotic way, sometimes hurting the majority groups and sometimes the minority groups, with no clear pattern.

In short, the authors show that there is no single "magic button" that fixes fairness, privacy, and accuracy all at once. The results suggest that the impact of these tools depends heavily on the size of the group, the type of computer model being used, and the specific method chosen. If you want to build a fair and private system, you can't just look at the average. You have to audit every single subgroup, because what works for the majority might be a privacy nightmare or a utility disaster for the minority. The paper doesn't claim to have solved the problem forever, but it provides the first clear map showing exactly where the traps are hidden in the forest of fairness and privacy.

Technical Summary: Auditing Fairness-Privacy Trade-offs at the Subpopulation Level

Problem Statement

Machine learning models deployed in sensitive domains (healthcare, law enforcement, finance) must satisfy utility, fairness, and privacy constraints simultaneously. While prior research has extensively examined how privacy-preserving mechanisms (specifically Differential Privacy) affect fairness, the inverse relationship remains underexplored: how do fairness-enhancing algorithms influence privacy leakage?

Existing evaluations typically operate at an aggregate level, obscuring disparities across subpopulations. In practice, risks are rarely uniform; minority or underrepresented groups may experience disproportionately poor fairness, weaker privacy protections, or severe utility losses. Furthermore, it is unclear whether widely used fairness interventions (pre-, in-, and post-processing) increase or decrease membership inference risks, and how these effects interact with Differential Privacy (DP) across different demographic subgroups.

Methodology

1. Experimental Framework

The authors conducted a comprehensive empirical study across ten variants of six real-world datasets (Bank Marketing, COMPAS, Law School Admissions, German Credit, Law School GPA, MEPS 19) and one synthetic dataset. The study utilized five classifier architectures: Decision Trees (DT), Random Forests (RF), Neural Networks (NN), Differentially Private Random Forests (DPRF), and Differentially Private Stochastic Gradient Descent Neural Networks (DP-SGD NN).

2. Fairness Interventions

The study evaluated five distinct fairness mitigation strategies across three categories:

Pre-processing: Disparate Impact Remover (DIR), Reweighing (REW), and Synthetic Mitigator (SYN).
In-processing: Exponentiated Gradient Reduction (EGR).
Post-processing: Calibrated Equalized Odds (CPP).

3. Privacy Auditing via Membership Inference Attacks (MIAs)

To quantify privacy risks, the authors adapted three state-of-the-art MIAs for subgroup-specific auditing:

Likelihood Ratio Attack (LiRA): Modified to restrict in-distribution and out-distribution sampling to specific fairness-relevant subpopulations. This allows for the estimation of subgroup-specific Gaussian parameters (mean and variance of confidence scores) to calculate likelihood ratios.
Optimal-Quantile-Threshold Attack (OQTA): Uses a separate population dataset to sweep quantiles and determine subgroup-specific thresholds.
Optimal-Threshold Attack (OTA): Applies threshold-based attacks on per-example losses within specific subgroups.

4. Theoretical Analysis

The paper provides a theoretical characterization of subpopulation privacy risk using influence functions. The authors derive that:

Small groups are more vulnerable: The membership signal scales as $O(1/n_g)$ , where $n_g$ is the subgroup size. Removing a sample from a small group eliminates a larger share of group-specific knowledge, amplifying the observable change in model predictions.
DP interaction: DP adds noise that reduces the signal-to-noise ratio. While this lowers MIA success, it disproportionately harms small groups where the learned signal is already weak, potentially causing catastrophic utility degradation.
Fairness mechanisms: Interventions alter MIA risk by changing decision boundaries, margins, and loss landscapes. For instance, Reweighing (REW) can improve margins for upweighted groups, reducing the distinguishability between members and non-members.

Key Contributions

First Systematic Subpopulation Audit: The paper provides the first comprehensive analysis of how pre-, in-, and post-processing fairness algorithms affect subgroup-level privacy risk and utility, moving beyond aggregate metrics.
Adaptation of LiRA for Subgroups: The authors extend the Likelihood Ratio Attack to enable fine-grained measurement of membership inference vulnerability across demographic groups, addressing statistical instability in small subgroups through robust averaging and cross-validation with non-parametric attacks.
Characterization of DP-Fairness Interactions: The study systematically characterizes how Differential Privacy interacts with fairness interventions, revealing that DP's privacy benefits and utility costs are unevenly distributed.
Unified Empirical Framework: The authors introduce a framework to jointly evaluate fairness, privacy, and utility at the subpopulation level, demonstrating that these objectives are not inherently in conflict but depend critically on model architecture, subgroup representation, and the chosen mitigation strategy.

Key Results

Disparate Vulnerability: Minority subpopulations consistently exhibit higher privacy leakage (higher MIA success rates) compared to majority groups across all datasets and models.
Impact of Fairness Algorithms:
- EGR (In-processing): Generally achieves the strongest reductions in privacy risk for complex models like Random Forests but performs poorly on Decision Trees and Neural Networks.
- REW and SYN (Pre-processing): Yield modest privacy-risk reductions for simpler models (DT, NN) but are dataset-dependent.
- DIR (Pre-processing): Exhibits inconsistent behavior, sometimes increasing and sometimes decreasing privacy risk.
- CPP (Post-processing): Consistently yields privacy risks near 0.5 (random guessing), suggesting strong privacy protection. However, it introduces uneven and unpredictable impacts on subgroup utility. In several cases, majority groups suffer significant accuracy drops while minority groups retain or improve accuracy, indicating a non-systematic redistribution of errors.
Fairness-Privacy-Utility Trade-offs:
- Fairness interventions do not uniformly increase privacy risk; the impact depends on model architecture and mitigation strategy.
- DP and Fairness: While DP reliably reduces privacy risk, its utility costs are highly uneven. Underrepresented groups often collapse to near-random accuracy under DP, whereas majority groups retain acceptable performance.
- Overall Accuracy vs. Subgroup Utility: While fairness algorithms preserve overall accuracy within ±1–3%, subpopulation-level utility shifts can be dramatic (up to ±33%).

Significance and Claims

The paper claims that fairness and privacy are not inherently conflicting objectives. Instead, their interaction is contingent upon specific factors:

Model Architecture: Different classifiers respond differently to fairness interventions regarding privacy leakage.
Subgroup Representation: The size and prevalence of subgroups in training data fundamentally shape vulnerability.
Mitigation Strategy: The choice of fairness technique (pre-, in-, or post-processing) dictates the direction and magnitude of privacy impact.

The authors emphasize that aggregate evaluations obscure these critical disparities. By introducing a unified empirical framework for subpopulation-level auditing, the work promotes a more nuanced understanding of the trade-offs in ML systems, suggesting that practitioners must evaluate fairness, privacy, and utility jointly at the granularity of subpopulations to ensure equitable and secure deployments.