No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification models

This paper empirically analyzes the distinct impacts of label and selection bias on classification model evaluation and performance using a new framework for introducing controlled bias, revealing that fairness-accuracy trade-offs disappear when models are evaluated on unbiased data and demonstrating that the effectiveness of mitigation methods depends on the specific bias type present.

The Gist

Here is an explanation of the paper, translated into simple language with creative analogies.

The Big Idea: You Can't Judge a Book by Its Biased Cover

Imagine you are a teacher trying to grade a class of students. However, there's a problem: the test papers you have been given to grade were tampered with before the students even saw them.

• Scenario A (Label Bias): Someone took a red pen and crossed out the "A" grades of the girls, changing them to "F"s, even though the girls actually did great work.
• Scenario B (Selection Bias): The teacher only collected test papers from the back row of the classroom, ignoring the front row entirely. Or worse, they only collected papers from students who volunteered to take the test, and the shy students (who might have studied harder) stayed home.

This paper argues that if you use these tampered papers to judge the students' abilities, you will get the wrong answer. Furthermore, if you try to "fix" the grading system using these bad papers, your fixes might make things worse.

The authors, Magali Legast, Toon Calders, and François Fouss, built a special laboratory to test this. They wanted to answer three questions:

1. How does tampering with data mess up our evaluation of AI models?
2. Does fixing the bias actually hurt the model's accuracy (the old "fairness vs. accuracy" debate)?
3. Which "fixes" work best for which specific type of tampering?

---

The Experiment: Building a "Time Machine" for Data

Usually, researchers take a real-world dataset (like a list of loan applications) and say, "This data is biased, let's try to fix it." The problem is, they don't know exactly how the bias got there, so they can't be sure if their fix actually worked.

The Authors' Solution:
They started with a dataset they believed was already fair (like a clean, honest record of student performance). Then, they acted like "data saboteurs." They artificially injected specific types of bias into the data to create a "distorted world."

• The Setup: They had a Fair World (the original clean data) and a Biased World (the tampered data).
• The Test: They trained AI models on the Biased World but tested them on the Fair World.
• The Result: This allowed them to see the true performance of the AI, rather than just how well it fit the broken data.

---

Key Findings (The "Aha!" Moments)

1. The "Fairness vs. Accuracy" Myth is Dead

For years, experts believed you had to choose: either your AI is accurate (predicts well) or it is fair (treats everyone equally), but you can't have both.

The Paper's Verdict: This trade-off is an illusion caused by bad testing.

• Analogy: Imagine a runner training on a track with a giant hill on one side. If you tell the runner to "run fast," they might lean into the hill, making them slower overall. If you tell them to "run fair" (ignore the hill), they might stumble.
• The Reality: When you test the runner on a flat, fair track (the unbiased test set), you realize that the runner who learned to ignore the hill actually runs the fastest and the fairest.
• Conclusion: When you evaluate models on fair data, you often find that you can improve fairness without losing accuracy. In fact, the most accurate models are often the fairest ones.

2. Not All "Fixes" Work for All "Breaks"

The paper tested eight different methods to fix bias (like "Reweighing," "Massaging," and "FTU"). They found that one size does not fit all.

• Analogy: Think of bias like different types of injuries.
  • Label Bias is like a broken leg. You need a cast (a specific fix).
  • Selection Bias is like a sprained ankle. You need a brace (a different fix).
  • If you try to put a cast on a sprained ankle, it won't help, and might even make it worse.

Specific Findings:

• Reweighing (giving more importance to underrepresented groups) worked great for Selection Bias (missing data), but was okay for Label Bias.
• Massaging (changing labels of people near the decision line) worked well for Label Bias (wrong labels), but made Selection Bias much worse.
• FTU (ignoring sensitive attributes like gender/race) worked surprisingly well across the board, but only if the other data features were strong enough to predict the outcome without needing to "peek" at the sensitive info.

3. The Danger of "Reverse Discrimination"

Some methods tried to fix bias by aggressively flipping the results to make the numbers look equal.

• Analogy: Imagine a scale that is tipped to the left. To fix it, you don't just add weight to the right; you start throwing heavy rocks off the left side. Eventually, the scale tips too far to the right, and now the other group is being treated unfairly.
• The Paper's Warning: Some methods, when applied to the wrong type of bias, actually created "reverse unfairness," hurting the privileged group to over-compensate for the bias.

---

Why This Matters for You

This paper is a wake-up call for anyone building or using AI.

1. Don't trust the test set: If you train an AI on biased data and test it on the same biased data, you are just confirming the bias. You need a "gold standard" or a "fair test" to see if the AI is actually doing a good job.
2. Know your enemy: Before you try to fix an AI, you need to know how it got broken. Was the data stolen? Were the labels wrong? Different problems need different solutions.
3. Fairness and Accuracy are friends: Stop thinking you have to sacrifice accuracy to be fair. If you measure things correctly, you'll find that the best models are usually both accurate and fair.

The Bottom Line

The authors built a "bias simulator" to prove that our current way of testing AI is flawed. They showed that if we stop using broken rulers to measure our tools, we can build AI systems that are not only smarter but also fairer, without having to make a painful trade-off between the two.

Technical Summary

Here is a detailed technical summary of the paper "No evaluation without fair representation: Impact of label and selection bias on the evaluation, performance and mitigation of classification models" by Legast, Calders, and Fouss.

1. Problem Statement

Machine learning models often suffer from biases (e.g., label bias and selection bias) that lead to discriminatory outcomes. However, the field of algorithmic fairness faces two critical limitations in how these issues are studied and mitigated:

1. Biased Evaluation: Most existing studies evaluate bias mitigation methods using the same biased dataset for both training and testing. This induces a "fairness-accuracy trade-off" (the belief that improving fairness inevitably reduces accuracy) and leads to misleading conclusions because the "ground truth" itself is distorted.
2. Unclear Bias Origins: Researchers often use standard datasets (e.g., Adult, COMPAS) where the source, type, and extent of bias are unknown or uncontrolled. This makes it difficult to determine which mitigation method works for which specific bias type.

The paper argues that without a "fair representation" (an unbiased test set representing the true underlying world), it is impossible to accurately evaluate model performance or the effectiveness of fairness interventions.

2. Methodology: The Biasing and Evaluation Framework

To address these issues, the authors propose a novel Biasing and Evaluation Framework based on the "Fair World" concept.

• Conceptual Model: The framework assumes the existence of a "Fair World" (an ideal distribution where fairness criteria hold). Real-world datasets are viewed as distorted versions of this world due to bias injection.
• Dataset Construction:
  • The authors start with real-world datasets (Student Performance, OULAD) that exhibit very low initial discrimination (acting as the "Fair World" proxy).
  • They artificially inject controlled bias into these datasets to create "Biased Worlds."
  • Bias Types Modeled:
    • Label Bias: Systematic penalty on the unprivileged group's scores before thresholding (e.g., lowering grades for a specific group).
    • Selection Bias: Undersampling of specific groups. Three subtypes were modeled:
      • Random Selection: Random removal of unprivileged individuals.
      • Self-Selection: Removal of unprivileged individuals based on lower scores (simulating lower application rates).
      • Malicious Selection: Removal of unprivileged individuals with positive labels and privileged individuals with negative labels (simulating active discrimination).
• Experimental Pipeline:
  1. Train models on the biased datasets (with and without mitigation).
  2. Evaluate models on the unbiased (fair) test set derived from the original data.
  3. Compare these "Fair Evaluations" against traditional "Biased Evaluations" (evaluating on the biased test set).
• Metrics: The study uses Accuracy, Statistical Parity Difference (SPD), Equalized Odds Difference (EqOd), Balanced Conditioned Consistency (BCC), and Generalized Entropy Index (GEI).
• Mitigation Methods Tested: Eight pre- and post-processing methods were evaluated, including Reweighing, Massaging, Fairness Through Unawareness (FTU), Equalized Odds Postprocessing (EOP), Calibrated Equalized Odds (CEO), and Reject Option Classifiers (ROC).

3. Key Contributions

1. Framework for Controlled Bias Injection: A reproducible framework to model specific bias types (label and selection) in real-world data, allowing for the creation of "dual-label" datasets (biased training, fair testing).
2. Refutation of the Fairness-Accuracy Trade-off: Empirical evidence showing that when models are evaluated on unbiased data, there is no inherent trade-off between fairness and accuracy. In many cases, mitigation methods improved both simultaneously.
3. Bias-Type Specificity: Demonstration that the effectiveness of a mitigation method is highly dependent on the specific type of bias present. A method effective for label bias may fail or even worsen performance under selection bias.
4. Evaluation Distortion Analysis: Quantification of how biased test sets skew metric measurements, often hiding the true performance of a model or falsely suggesting a trade-off.

4. Key Results

A. Impact of Evaluation Bias

• Metric Distortion: Evaluating on biased data produces misleading results. For instance, label bias skews metrics relying on ground truth (Accuracy, EqOd, GEI), while selection bias skews almost all metrics.
• False Trade-offs: Methods that appear to fail on biased test sets (showing low accuracy or high unfairness) often perform excellently on fair test sets. Conversely, methods that look successful on biased sets may actually be reinforcing bias when viewed against the fair ground truth.

B. Impact of Bias on Model Performance (Unmitigated)

• Label Bias: Has a detrimental effect on accuracy, group fairness, and individual fairness. It directly corrupts the relationship between features and the target label.
• Selection Bias: The impact is more nuanced and often negligible if:
  • The dataset is large.
  • The features are sufficiently predictive.
  • The "We Are All Equal" (WAE) assumption holds (groups are similar in the fair world).
  • Exception: Selection bias becomes significant when features are weak (complex tasks) or when the bias intensity is extreme.

C. Performance of Mitigation Methods

The study found that no single method works for all bias types.

• Reweighing: Highly effective for Selection Bias (both random and malicious) as it counteracts the sampling distortion. It is less effective for Label Bias.
• Massaging & ROC-SPD: Effective for Label Bias (by flipping labels to satisfy statistical parity) but harmful for Selection Bias. In selection bias scenarios, they introduce "reverse" unfairness by altering accurate labels to satisfy a biased statistical parity goal.
• FTU (Fairness Through Unawareness): Surprisingly robust across all scenarios, provided the sensitive attribute is not highly correlated with other features in the fair world.
• EOP/CEO (Equalized Odds based): Vulnerable to both label and selection bias. They often fail to recover the fair distribution and can degrade accuracy.
• General Finding: Methods designed to optimize a specific metric (like SPD) on biased data often fail to generalize to the fair world if the bias type distorts that metric's calculation.

5. Significance and Implications

• Paradigm Shift in Evaluation: The paper argues that the field must move away from using biased test sets as ground truth. To truly assess fairness, models must be evaluated against an unbiased representation of the world (even if synthetic or controlled).
• Context-Aware Mitigation: Practitioners cannot blindly apply a "best" fairness algorithm. They must first diagnose the type of bias in their data. For example, using Massaging on a dataset suffering from selection bias will likely make the problem worse.
• Debunking the Trade-off: The "Fairness-Accuracy Trade-off" is largely an artifact of evaluating models on biased data. When the ground truth is fair, it is possible to achieve high accuracy and high fairness simultaneously.
• Future Directions: The authors call for the creation of more "dual-label" datasets and the development of mitigation methods that are robust to unknown or mixed bias types, rather than methods optimized for specific, known distortions.

In conclusion, the paper establishes that fair representation is a prerequisite for fair evaluation. Without it, the field risks optimizing for the wrong objectives and misidentifying the efficacy of fairness interventions.