An Experimental Study on Fairness-aware Machine Learning for Credit Scoring Problems

This paper presents a comprehensive experimental study demonstrating that fairness-aware machine learning models achieve a superior balance between predictive accuracy and fairness compared to traditional classification models in the context of credit scoring.

The Gist

Imagine you are a bank manager. In the old days, if someone wanted a loan, you'd sit them down, look at their paperwork, maybe ask a few questions, and use your gut feeling to decide if they were a "good" risk or a "bad" risk. It was slow, it took a lot of coffee breaks, and it relied heavily on human judgment.

Today, banks use Machine Learning (ML) to do this instantly. Think of ML as a super-fast, super-smart robot assistant that reads thousands of financial records in a split second to predict who will pay back their loan and who won't. This is called Credit Scoring.

But here's the problem: Robots can be biased.

If you train a robot on historical data where, say, women were unfairly denied loans in the past, the robot might learn that "being a woman" means "don't give a loan." It's not being evil; it's just copying the mistakes of the past. This is the "unfairness" the paper talks about.

The Mission: Fixing the Robot's Brain

The authors of this paper asked a big question: "We have all these fancy new tools to make these robots fair, but do they actually work in the real world of banking?"

They decided to run a massive experiment. They didn't just talk about theory; they put these tools to the test.

The Ingredients of the Experiment

1. The Test Subjects (The Datasets)
They gathered five different "playgrounds" of real-world financial data. Imagine these as five different neighborhoods with different types of people. Some neighborhoods had data on credit card users, others on German applicants, and others on clients from Central Asia. They checked these neighborhoods to see if the data itself was already biased (like finding that the neighborhood records already treated men and women differently). They found that, yes, the data was often biased to begin with.

2. The Tools (The Models)
They tested three different ways to "fix" the robot:

• Pre-processing (Cleaning the Ingredients): Before the robot even starts cooking, you wash and sort the vegetables. You remove the bad stuff or balance the ingredients so the robot doesn't start with a bias.
  • Analogy: Like a chef tasting the soup before adding salt to make sure it's not too salty.
• In-processing (Training with Rules): You teach the robot a new rule while it's learning. "Hey, when you make a decision, you must treat men and women exactly the same, even if it's slightly harder to get the answer right."
  • Analogy: Like a teacher telling a student, "You can't just guess; you have to follow this specific fair rule to get an A."
• Post-processing (Adjusting the Result): The robot makes its decision, and then a human (or another program) looks at the result and tweaks it. "Wait, you denied this woman, but based on the rules, you should have approved her. Let's flip the switch."
  • Analogy: Like a referee blowing the whistle after a play to say, "That was a foul, let's restart."

3. The Scorecard (Fairness Measures)
How do you know if the robot is actually fair? You can't just look at it. You need a ruler. The paper tested about 8 different "fairness rulers."

• Statistical Parity: Does the robot approve men and women at the same rate?
• Equal Opportunity: If a man and a woman are both good borrowers, does the robot give them both a loan?
• Predictive Parity: If the robot says someone is a "good" borrower, is that prediction equally accurate for both men and women?

What Did They Find? (The Results)

The experiment was like a race between traditional robots (the old, biased ones) and the new "Fairness-Aware" robots.

• The Old Way: The traditional models were often very accurate at predicting who would pay back, but they were often unfair. They might accidentally discriminate against a specific group.
• The "Fair" Way: The new models did a great job of being fair. They significantly reduced the discrimination.
• The Trade-off: Usually, when you make something fair, it gets a little less accurate (like a referee being so strict they miss a real goal). However, the paper found that the In-processing method (specifically a tool called AdaFair) was the "Goldilocks" solution. It managed to be both highly accurate and very fair. It didn't have to sacrifice much accuracy to be fair.

The Big Winner: The AdaFair model was the star of the show. It consistently balanced the need to make money (accuracy) with the need to be just (fairness) better than the other methods across all five different neighborhoods (datasets).

The Takeaway

This paper is a reality check for the banking world. It says:

1. Bias is real: Our data is flawed, so our robots will be flawed unless we fix them.
2. Tools exist: We have the technology to fix this.
3. It works: We don't have to choose between making money and being fair. With the right tools (like AdaFair), we can do both.

In simple terms: The authors proved that we can teach our banking robots to be not just smart, but also kind and fair, without making them stupid. It's about building a financial system where everyone gets a fair shot, regardless of who they are.

Technical Summary

Here is a detailed technical summary of the paper "An Experimental Study on Fairness-aware Machine Learning for Credit Scoring Problems."

1. Problem Statement

The digitalization of credit scoring has become critical for financial institutions, relying heavily on Machine Learning (ML) to evaluate customer creditworthiness. However, standard ML models often exhibit bias against protected attributes (e.g., race, gender), leading to discriminatory lending decisions. While numerous fairness-aware ML models and metrics exist, there is a lack of comprehensive empirical research evaluating their performance specifically within the credit scoring domain.

Key challenges identified include:

• Trade-off: Balancing predictive accuracy with fairness constraints.
• Metric Selection: Over 20 different fairness measures exist, but no single metric is universally applicable to all scenarios.
• Data Bias: Real-world financial datasets often contain inherent biases that propagate through standard classification models.

2. Methodology

The authors conducted a comprehensive experimental study involving data analysis, model implementation, and rigorous evaluation across multiple dimensions.

A. Datasets

Five widely used public credit scoring datasets were selected based on criteria including the presence of protected attributes, a target classification label, and a minimum of 500 instances:

1. Credit Approval (Australian): 690 instances; Protected attributes: Sex, Age.
2. Credit Card Clients (Taiwan): 30,000 instances; Protected attributes: Sex, Education, Marital status.
3. Credit Scoring (FinTech/Central Asia): 8,755 instances; Protected attributes: Age, Sex, Marital status.
4. German Credit: 1,000 instances; Protected attributes: Age, Sex.
5. PAKDD 2009 Credit Risk: 50,000 instances; Protected attributes: Age, Sex, Marital status.

Bias Analysis: The authors utilized Bayesian Networks (BN) to analyze the structural relationships between protected attributes and the class labels. The analysis confirmed that direct or indirect connections exist between protected attributes (like Sex) and credit outcomes in all datasets, indicating inherent bias.

B. Models Evaluated

The study compared Traditional Classifiers against three categories of Fairness-aware ML models:

1. Traditional Baselines: Decision Tree (DT), Naive Bayes (NB), Multi-layer Perceptron (MLP), k-Nearest Neighbors (kNN).
2. Pre-processing Approaches:
   • Learning Fair Representations (LFR): Encodes data to obscure protected attributes.
   • Disparate Impact Remover (DIR): Adjusts feature values to enhance group fairness.
3. In-processing Approaches:
   • Agarwal's Method: Reduces fair classification to cost-sensitive classification with constraints.
   • AdaFair: An adaptive boosting algorithm that optimizes for cumulative fairness and class imbalance.
4. Post-processing Approaches:
   • Equalized Odds Post-processing (EOP): Modifies output labels via linear programming.
   • Calibrated Equalized Odds Post-processing (CEP): Optimizes calibrated scores to achieve equalized odds.

C. Evaluation Metrics

The study evaluated models using:

• Predictive Performance: Accuracy and Balanced Accuracy (BA) to account for class imbalance.
• Fairness Measures: Eight distinct metrics were calculated:
  • Statistical Parity (SP)
  • Equal Opportunity (EO)
  • Equalized Odds (EOd)
  • Predictive Parity (PP)
  • Predictive Equality (PE)
  • Treatment Equality (TE)
  • Absolute Between-ROC Area (ABROCA)
  • Note: Lower values for fairness metrics indicate less discrimination (0 is ideal).

3. Key Contributions

1. Comprehensive Benchmarking: Provided a large-scale evaluation of 6 fairness-aware models (across pre-, in-, and post-processing) against traditional baselines on 5 diverse financial datasets.
2. Bias Detection: Demonstrated the use of Bayesian Networks to systematically identify and visualize bias structures within credit datasets prior to modeling.
3. Metric Analysis: Evaluated the performance of models across 8 different fairness notions, highlighting the difficulty of satisfying multiple fairness criteria simultaneously.
4. Performance Trade-off Analysis: Quantified the trade-offs between accuracy and fairness, identifying specific models that offer the best balance for credit scoring.

4. Experimental Results

The results varied by dataset and model type, but several consistent trends emerged:

• In-processing Superiority: AdaFair consistently demonstrated the best overall performance. It achieved high predictive accuracy (often matching or exceeding traditional models) while simultaneously maintaining strong fairness scores across multiple metrics. For example, on the Credit Scoring dataset, AdaFair achieved 99.43% accuracy with low fairness metric values.
• Pre-processing Limitations: While models like LFR (Learning Fair Representations) could achieve near-perfect fairness scores (e.g., 0.0 on SP, EO, EOd), they often suffered from a drastic drop in predictive accuracy and balanced accuracy (sometimes falling below 0.5), rendering them impractical for real-world lending.
• Post-processing Constraints: Post-processing methods (EOP, CEP) were effective at adjusting outcomes but struggled to calculate ABROCA (due to threshold requirements) and generally did not outperform in-processing methods in balancing accuracy and fairness.
• Traditional Models: Traditional models (DT, NB, MLP) generally offered high accuracy but exhibited significantly higher discrimination (higher fairness metric values) compared to fairness-aware counterparts.
• Dataset Specifics:
  • On the German Credit dataset, traditional Naive Bayes had the highest accuracy, but LFR-MLP achieved perfect fairness scores at the cost of low balanced accuracy (0.5).
  • On the PAKDD dataset, all methods showed lower balanced accuracy, but AdaFair still outperformed others on 6 out of 9 evaluation measures.

5. Significance and Conclusion

• Practical Implication: The study proves that fairness-aware ML is not merely a theoretical exercise but can be practically applied to credit scoring to reduce discrimination without sacrificing predictive power. AdaFair is highlighted as a robust candidate for deployment.
• Methodological Insight: The research underscores that "one size fits all" does not apply to fairness. Different datasets and fairness definitions require different approaches. Pre-processing often sacrifices too much accuracy, while in-processing offers a better equilibrium.
• Future Directions: The authors identify limitations, including the evaluation of single protected attributes (ignoring intersectionality) and the reliance on existing datasets. Future work aims to develop fair synthetic data generation and explainable AI (XAI) to understand the root causes of bias within algorithms and data.

Code Availability: The source code and datasets used in the study are publicly available at https://github.com/tailequy/faircredit.