📊 statistics

Ablation Study of Class Imbalance Techniques for Credit Default Prediction with SHAP-based Explainability Analysis

This study demonstrates that on credit default prediction tasks, XGBoost's built-in `scale_pos_weight` parameter significantly outperforms the traditional SMOTE resampling technique in recall, highlighting that the optimal strategy for handling class imbalance depends on the specific classifier used rather than a universal resampling method.

Original authors: Atharv Tiwari

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Atharv Tiwari

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find a single thief hiding in a crowd of 100 people. Ninety-two of them are innocent, and only eight are guilty. If you just guess "everyone is innocent," you'd be right 92% of the time! But you'd catch zero thieves. In the world of computer science, this is called the "class imbalance" problem. It happens when a computer model is trained on data where one outcome (like a loan being paid back) happens way more often than the other (a loan defaulting). Because the computer sees so many "good" examples, it gets lazy and just predicts "good" for everyone, failing to spot the rare "bad" ones.

For years, the standard fix for this has been a technique called SMOTE. Think of SMOTE as a photocopier for the rare cases. If you only have eight photos of thieves, SMOTE creates fake, synthetic photos of new thieves by mixing and matching details from the real ones, hoping to give the computer enough practice to learn the pattern. But there's another way. Some computer models, specifically a powerful type called XGBoost, have a built-in "volume knob" for mistakes. Instead of making up fake data, you can just tell the computer, "Hey, if you miss a thief, that's a huge mistake! If you miss an innocent person, that's a small mistake." This forces the model to pay extra attention to the rare cases without ever touching the original data.

The big question is: Is it better to make up fake practice data (SMOTE), or just turn up the volume on the mistakes (the built-in knob)? A new study by Atharv Tiwari from Punjab Engineering College puts these two approaches head-to-head to see which one actually catches more thieves in the real world of credit scoring.

The Great Credit Card Showdown

In this study, the researcher set up a massive experiment using a real-world dataset of over 300,000 loan applications. In this crowd, 92% of people paid their loans back, and only 8% defaulted. It was the perfect setup to test how well different computer models could spot the 8% without getting distracted by the 92%.

The researcher didn't just pick one model; they tested three different "detectives":

Logistic Regression: The old-school, reliable detective who follows strict rules.
Random Forest: A team of detectives who each look at the clues differently and vote on the answer.
XGBoost: A super-smart detective that learns from its mistakes one by one, getting sharper with every round.

Each detective was given three different training strategies:

The "Do Nothing" Strategy: Just throw the raw data at them and see what happens.
The "Volume Knob" Strategy: Use the model's built-in settings to penalize missing a defaulter heavily (no fake data).
The "Photocopier" Strategy: Use SMOTE to create fake defaulters to balance the training data.

The Shocking Result

The results were a bit like a plot twist in a mystery movie. The "Do Nothing" strategy failed miserably for everyone, catching almost no defaulters (less than 2.2% recall). That was expected. But the battle between the "Volume Knob" and the "Photocopier" was where things got interesting.

When the researcher used XGBoost, the built-in "Volume Knob" strategy crushed the competition. It managed to catch 64.8% of the actual defaulters. The "Photocopier" strategy (SMOTE), which has been the go-to solution for decades, only caught 43.3%. That is a massive gap of 21.5 percentage points. The researcher ran strict statistical tests (called McNemar's test) and confirmed this wasn't a fluke; the difference was so huge it was statistically certain ( $p < 10^{-6}$ ).

However, the story changed completely when they switched to Random Forest. For this model, the "Volume Knob" barely worked (catching only 7.8% of defaulters), while the "Photocopier" (SMOTE) did much better, catching 41.2%.

This suggests that there is no single "magic bullet" for fixing imbalanced data. What works for one type of computer model might fail for another. The study suggests that for the specific, powerful XGBoost model, simply telling the model to care more about the rare mistakes is far superior to feeding it a diet of synthetic, fake data.

Why the Photocopier Failed (and the Knob Won)

So, why did making up fake data hurt the XGBoost model? The researcher explains that when you have a complex dataset with 192 different features (like income, age, and credit history), creating a fake person by mixing two real people can result in a "monster" that doesn't exist in reality. It's like trying to blend a picture of a cat and a dog to make a new animal; the result might look weird and confuse the computer. The model ends up learning from noise instead of real patterns.

The built-in "Volume Knob" avoids this trap entirely. It doesn't touch the data or create fake examples. It just changes the math behind the scenes to say, "Don't miss the rare ones!" This kept the training clean and allowed XGBoost to perform at its best.

The "Why" Behind the "Who"

To make sure this winning model wasn't just a black box making random guesses, the researcher used a tool called SHAP (which stands for SHapley Additive exPlanations). Think of SHAP as a magnifying glass that shows exactly which clues the detective used to make a decision.

The analysis revealed that the model was behaving logically. The most important clue was the external bureau score (a credit score from other banks). This makes perfect sense: if someone has a history of paying back other lenders, they are likely to pay you back too. The model also looked at the ratio of the loan amount to the repayment amount and the borrower's age. Crucially, the model's reasoning was transparent and defensible, which is a big deal for banks that need to explain their decisions to regulators.

The Bottom Line

The study concludes that before you start generating thousands of fake data points to fix an imbalance, you should try the simpler, built-in tools your model already has. For XGBoost, the built-in weighting was not just easier to use; it was significantly more effective, catching nearly 22% more defaulters than the traditional method.

However, the researcher warns that this isn't a universal rule. Because Random Forest performed better with the "Photocopier" method, you can't assume one technique works for every model. The best approach depends entirely on which "detective" you are using. In the regulated world of banking, where catching defaulters is vital but explaining decisions is mandatory, this finding offers a clearer, more efficient path forward: try the simple knob first, but always check if it fits your specific model.

Technical Summary: Ablation Study of Class Imbalance Techniques for Credit Default Prediction with SHAP-based Explainability Analysis

Problem Statement
Credit default prediction datasets, such as the Home Credit Default Risk dataset, exhibit severe class imbalance (approximately 92% non-defaulters vs. 8% defaulters). In such scenarios, standard accuracy metrics are misleading, as a model predicting "no default" for all applicants achieves high accuracy but zero utility. The standard industry response since 2002 has been the Synthetic Minority Over-sampling Technique (SMOTE), which generates synthetic minority examples to rebalance the training set. However, it remains untested whether gradient boosting models, specifically XGBoost, can handle this imbalance natively through loss function adjustments (via the scale_pos_weight parameter) without introducing synthetic data or potential data leakage. Furthermore, existing literature often fails to integrate class imbalance handling with model explainability, a critical requirement for regulatory compliance in finance.

Methodology
The study conducts a controlled ablation study across three classifiers—Logistic Regression, Random Forest, and XGBoost—evaluated on 307,511 loan applications with 122 original features (expanded to 192 after engineering and preprocessing).

Experimental Design: Nine configurations were tested, combining three classifiers with three imbalance strategies:
1. Baseline (A): No rebalancing; default hyperparameters.
2. Native Reweighting (B): Built-in class weighting (class_weight='balanced' for RF/LogReg; scale_pos_weight for XGBoost).
3. SMOTE + Threshold Tuning (C): SMOTE applied to the training set to achieve class parity, followed by threshold optimization on the precision-recall curve to maximize F1.
Evaluation: Performance was measured using Recall, Precision, F1-score, and ROC-AUC. Statistical significance was determined using McNemar's exact test on pairwise comparisons. Stability was verified via five-fold stratified cross-validation.
Explainability: The best-performing configuration was analyzed using SHAP (SHapley Additive exPlanations) to decompose predictions into per-feature contributions, ensuring the model's logic aligns with regulatory transparency requirements.

Key Results
The study yielded distinct findings regarding the interaction between model architecture and imbalance handling:

XGBoost Superiority with Native Weighting: XGBoost utilizing scale_pos_weight (Strategy B) achieved a recall of 64.8%, significantly outperforming the SMOTE approach (Strategy C), which achieved 43.3%. The 21.5 percentage point gap was statistically significant ( $p < 10^{-6}$ ).
Model-Dependent Efficacy: The results were not uniform across all models. While native weighting excelled in XGBoost and Logistic Regression, it performed poorly in Random Forest (7.8% recall). Conversely, Random Forest performed best with SMOTE (41.2% recall). This demonstrates that imbalance handling is not a "one-size-fits-all" solution but depends heavily on the classifier family.
Stability: Five-fold cross-validation confirmed the stability of the XGBoost native weighting approach, with a mean recall of 64.1% ( $\pm 0.7\%$ ) and ROC-AUC of 0.755 ( $\pm 0.003$ ).
Feature Importance: SHAP analysis revealed that EXT_SOURCE_AVG (aggregated external bureau scores) was the dominant driver of risk assessment. The model's reasoning was monotonic and defensible: lower bureau scores increased the probability of default. Notably, gender (CODE_GENDER_M) emerged as the third most important feature, identified autonomously by the model.

Significance and Claims
The paper claims three primary contributions to the field of credit risk modeling:

Challenging the SMOTE Default: The study provides empirical evidence that for XGBoost, native loss function adjustment (scale_pos_weight) is superior to SMOTE on high-dimensional credit data. The authors argue that SMOTE may introduce noise in high-dimensional spaces by interpolating between minority instances that differ across many axes, whereas native weighting avoids data modification entirely.
Contextualizing Imbalance Strategies: The research highlights that the efficacy of an imbalance technique is contingent on the underlying algorithm. A strategy validated on one model family (e.g., SMOTE for Random Forest) does not necessarily transfer to another (e.g., XGBoost), urging practitioners to validate strategies per model rather than assuming universality.
Integration of Explainability: By coupling high-performance imbalance handling with SHAP analysis, the study addresses the "Gap 3" where imbalance research often lacks the interpretability layer required for regulatory deployment (e.g., under RBI Draft Guidance). The proposed stack (XGBoost + scale_pos_weight + SHAP) offers a deployable solution that requires no synthetic data pipelines, reduces data leakage risks, and provides transparent decision logic.

Limitations
The authors acknowledge that findings are based on a single public dataset and may not generalize to proprietary banking data with different feature distributions. Additionally, the study did not incorporate business-specific misclassification costs, fairness analysis across demographic groups, or advanced loss functions like focal loss. The threshold calibration for SMOTE was not nested within cross-validation, potentially leading to slightly optimistic performance estimates for that specific strategy.