📊 statistics

How Reliable are Fairness Audits with Unreliable Data?

This paper introduces a seed-calibrated stress test to demonstrate that protected-label missingness in fairness audits often does not significantly alter mitigation outcomes beyond natural seed variability, though it can expose specific failure modes like intersectional harm during threshold optimization, suggesting that missingness effects should be reported with careful calibration and context rather than dismissed as audit fragility.

Original authors: Yash Vardhan Tomar

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yash Vardhan Tomar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to decide which of five different "fairness recipes" works best for a machine learning model. Your goal is to make sure the model treats different groups of people (like different races or genders) fairly.

To do this, you need a Fairness Audit. This is like a taste test where you check the model's predictions against real data to see if it's biased. But here's the catch: sometimes, the data is missing the "ingredients" you need to know who belongs to which group. Maybe people didn't fill out the form, or the records got lost.

This paper asks a simple but crucial question: If we are missing some of these group labels, can we still trust our audit results, or is the audit just breaking because the data is messy?

Here is the breakdown of what the researchers found, using some everyday analogies.

1. The "Noise" Problem: Is it the Missing Data or Just Randomness?

Imagine you are trying to pick the best runner in a race.

The Problem: Sometimes, the stopwatch is slightly off, or the runners start a split second later than others. Even if you have perfect data, if you run the race twice with different random starting conditions (called "seeds" in the paper), you might pick a slightly different winner each time.
The Paper's Insight: The researchers realized that when we see an audit change its mind because data is missing, we often panic and think, "Oh no, the missing data broke everything!"
The Reality Check: They found that missing data doesn't actually break the audit as much as we think. In fact, the audit changes its mind just as often (or even more often) when it has perfect data, simply because of random noise.
The Analogy: It's like a judge changing their mind about the best runner just because they blinked at the wrong time. The paper says: "Don't blame the missing data yet. First, check if the judge is just being fickle." They created a "calibration" tool to separate the "fickle judge" noise from the actual damage caused by missing data.

2. The "No-Data" Cliff Edge

There is a specific point where the audit does get confused: when zero group labels are available.

What Happens: If you have no idea who is in which group, several different fairness recipes all end up doing the exact same thing (they all act like the standard, non-fair version).
The Result: When the audit has to pick a winner among these identical recipes, it just picks one randomly (like flipping a coin or picking the one that comes first alphabetically). This makes the audit look unstable, but it's actually just a tie-breaking issue, not a fundamental failure of the method.

3. The "Hidden Trap": The Intersectional Danger

This is the most important finding. Imagine you are checking if a school is fair.

The Trap: You check if the school is fair for "Boys" and fair for "Girls" separately. You find a recipe that makes things fair for both groups individually. You celebrate!
The Reality: But what about "Black Girls"? Or "Older Men"? The paper found that some methods (specifically one called Threshold Optimization) can make the single groups look great while secretly making things worse for the specific combinations of groups (the intersections).
The Analogy: It's like a diet plan that helps you lose weight on your arms and legs, but secretly makes you gain a dangerous amount of weight in your stomach. The "single-axis" audit says, "Great job!" but the "intersectional" audit says, "You're actually hurting the most vulnerable people."
The Finding: The researchers found that this "hidden harm" happens mostly when using the Threshold Optimization method. Even if the audit says the model is fair, this specific method often hides a sharp drop in accuracy for the most vulnerable subgroups.

4. The "Stress Test" Results

The researchers ran their audit on real-world data (like predicting income or employment) and simulated missing data in two ways:

Randomly Missing: Like a form getting lost in the mail (unrelated to who you are).
Systematically Missing: Like a specific group of people refusing to answer the question.

The Verdict:

Good News: As long as you have some data (even 20% or 40%), the audit recommendations usually stay stable. They don't flip-flop wildly just because data is missing.
Bad News: The real danger isn't the missing data itself; it's choosing the wrong fairness recipe. If you pick the "Threshold Optimization" recipe, you might think you've solved the fairness problem, but you've actually created a hidden trap for intersectional groups.

Summary: What Should You Take Away?

If you are running a fairness audit:

Don't panic over missing data immediately. First, check if your audit is just "noisy" (changing its mind randomly).
Watch out for the "Hidden Trap." Just because a model looks fair for Race A and Gender B doesn't mean it's fair for "Race A + Gender B."
Be careful with "Threshold Optimization." This specific method is great at fixing single-group problems but is very good at hiding harm to complex, intersectional groups.
Report the whole picture. Don't just say "We picked Method X." You need to report how the method behaves when data is missing, and check if it's hurting the most vulnerable subgroups.

In short: Missing data is annoying, but picking the wrong fairness tool is dangerous. The paper gives us a better way to tell the difference.

Technical Summary: How Reliable are Fairness Audits with Unreliable Data?

Problem Statement

Fairness audits are essential for responsible machine learning deployment, yet their reliability under protected-label missingness remains poorly understood. In practice, demographic attributes (e.g., race, sex) are often missing due to non-collection, user refusal, or record inconsistencies. When these labels are absent, auditors cannot accurately compute group-level disparities or select appropriate mitigation methods.

The central challenge addressed is distinguishing between audit instability caused by missing data and instability inherent to the audit process itself (e.g., seed-to-seed variation, near-equivalent candidate methods, and deterministic tie-breaking). Without a baseline for "ordinary" audit noise, a change in recommendation due to missing labels may be misinterpreted as a critical failure of the audit framework rather than a reflection of standard stochastic variance.

Methodology

The authors propose a seed-calibrated stress test to isolate the effects of missingness from inherent audit noise. The methodology involves:

Experimental Setup:
- Tasks: Four binary classification tasks from the ACS/Folktables dataset (Income, Employment, Public Coverage, Mobility).
- Protected Views: Race (RAC1P), Sex (SEX), and their intersection (RAC1P×SEX).
- Candidate Methods: Empirical Risk Minimization (ERM), Feature Mixing, Reweighing, Protected-Group Mixing, and Equalized-Odds Threshold Optimization.
- Selection Criteria: Accuracy-constrained Equalized Odds (EO), Demographic Parity (DP), Worst-Group Accuracy (WGA), and Intersectional EO.
Missingness Regimes:
- Complete-Label Oracle ( $r_0$ ): All protected labels are available.
- Missing Completely at Random (MCAR): Labels are hidden at rates of 80%, 60%, 50%, 40%, 20%, 10%, and 0%.
- Missing Not at Random (MNAR): Matched to specific MCAR targets (50%, 20%, 10%) with asymmetric label access (majority groups retain higher availability).
- Evaluation: Labels are always retained for the evaluation phase to ensure metrics are computed against ground truth; missingness only affects training and post-processing mitigation.
Calibration Mechanism:
- Seed-Pair Null: The audit is rerun with complete labels using different random seeds ( $s, s'$ ). The frequency of recommendation flips between these seeds establishes a baseline "noise floor."
- Matched Excess Flip Rate ( $\Delta_c(r)$ ): The difference between the flip rate observed under missingness and the seed-pair null. A positive $\Delta$ indicates missingness causes instability beyond normal audit variance.
- Hidden-Regression Screen: Checks if a method improves single-axis fairness (e.g., race or sex) while worsening intersectional outcomes (e.g., race×sex) or Worst-Group Accuracy (WGA).
Validation: A secondary experiment repeats the audit using a Random Forest base learner to test robustness against higher-variance models.

Key Results

1. Missingness Often Stays Below the Seed Null

Contrary to the intuition that missing labels destabilize audits, the study finds that positive-availability missingness usually does not move selected mitigation methods beyond the complete-label seed floor.

Raw Flip Rates: 37.3% of missing-label comparisons resulted in a different method selection.
Seed-Null Baseline: 53.8% of complete-label seed-pair comparisons resulted in a flip.
Calibrated Finding: After calibration, most missingness regimes (MCAR and MNAR with >0% availability) fall below the seed noise floor. Only the 0% (no-label) endpoint exceeds the seed floor (+6.3 percentage points), primarily due to deterministic tie-breaking among ERM-equivalent candidates.

2. Candidate Set Composition Matters

The "flip rate" is highly sensitive to the candidate set. When the audit is restricted to {ERM, Reweighing, Threshold Optimization}, the raw flip rate drops to 33.8% against a 39.8% seed null. This suggests that apparent fragility is often an artifact of including near-equivalent methods that trigger tie-breaking rules rather than a fundamental failure of the audit logic.

3. Threshold Optimization Drives Intersectional Harm

The most significant reliability failure identified is hidden intersectional regression, where single-axis fairness gains mask subgroup harm.

Overall Hidden-Regression Rate: 69.9% (calibrated against a 55.0% seed null).
Driver: Threshold Optimization is the primary driver, with a hidden-regression rate of 94.2%. It frequently improves single-axis EO metrics while causing severe drops in Worst-Group Accuracy (median drop of 12.34 pp).
Other Methods: Reweighing and Feature Mixing show lower hidden-regression rates (29.6% and 36.2%, respectively) and remain below the seed null when threshold optimization is excluded.

4. Robustness to Base Learner Variance

Validation with a Random Forest (higher variance than Logistic Regression) confirmed the main findings:

Strict flip rates increased slightly (37.3% to 41.1%), but the relative ordering of missingness effects remained consistent.
The hidden-regression pattern driven by threshold optimization persisted (93.5% rate), confirming that the failure mode is method-specific rather than model-specific.

Significance and Claims

The paper argues that protected-label missingness should not be treated as evidence of audit fragility without proper calibration. The authors claim:

Need for Seed-Null Calibration: Raw sensitivity analyses (e.g., "the method changed when labels were hidden") are misleading. Audits must report whether the change exceeds the variation inherent to the audit process under complete labels.
Contextual Reporting: Audit reports must include:
- Seed-null calibration: To distinguish missingness effects from noise.
- Candidate-set context: To account for tie-breaking artifacts.
- Intersectional consequences: To detect hidden regressions where single-axis gains hide intersectional harm.
Specific Failure Modes: The study identifies threshold optimization as a high-risk method that can produce "above-null" hidden regressions, particularly when labels are sparse.
Actionable Availability Curves: Audits should visualize how methods behave across a spectrum of label availability, separating recommendation changes from metric degradation.

In summary, the work provides a diagnostic framework to prevent the misinterpretation of missing data effects, emphasizing that many observed instabilities are artifacts of standard audit variance or specific method behaviors (like threshold optimization) rather than a general inability to audit under missingness.