Benchmarking Gradient Boosting and Explainable AI for Credit Default Prediction on Imbalanced Financial Data: A Leakage-Safe Multi-Seed Evaluation with SHAP and LIME

Every time a person applies for a credit card or a small loan, a hidden calculation happens in the background. Banks and lenders must decide whether to trust that person with money, a decision that hinges on predicting if they will fail to pay it back. This is not just a guess; it is a high-stakes financial judgment that determines who gets access to the economy and at what cost. To make these decisions, institutions rely on computer models that scan a person's history—their income, their past bills, and how they have handled debt before. These models must be accurate, but they also must be fair and transparent. Laws in the United States require that if a loan is denied, the lender must explain exactly why. This means the computer cannot just say "no"; it must point to the specific reasons, such as a late payment or a high balance, that led to the decision. If the model is a black box that no one understands, it creates legal risks and unfair outcomes for borrowers.

For years, researchers have tried to build better models to predict these defaults, often using complex artificial intelligence. However, a new study suggests that many of these previous attempts have been built on shaky ground. The researchers found that common practices in testing these models often introduce methodological errors, making the computers look smarter than they really are. They also discovered that some popular tricks used to fix data problems are actually unnecessary. By setting up a stricter, more honest way to test these systems, the team identified which models truly work best and how to explain their decisions without confusion. Their work provides a new, reliable standard for how financial institutions should evaluate the tools that decide who gets credit.

The study focused on a massive collection of real-world data from a credit card company in Taiwan, involving thirty thousand customers. About twenty-two percent of these customers eventually failed to pay their bills, creating a situation where the "default" cases were fewer than the "good" cases. This imbalance is common in finance and has led many researchers to use a technique called synthetic oversampling. This method creates fake data points to balance the numbers, hoping it will help the computer learn better. The researchers in this study, however, tested whether this extra step was actually needed. They built a rigorous testing protocol that kept the data strictly separated, ensuring that the computer never saw the test answers while it was learning. They ran the tests ten different times with different random starting points to make sure the results were stable and not just a lucky fluke.

When they compared six different types of computer models, the results were clear. The most powerful tools were a family of models known as gradient boosting ensembles. These are systems that combine many simple decision trees to make a single, highly accurate prediction. Three specific versions of these models—XGBoost, LightGBM, and CatBoost—performed almost identically, forming a top tier that significantly outperformed older, simpler methods like logistic regression and newer deep learning approaches. The top models correctly ranked the risk of default about seventy-nine percent of the time, a figure that was statistically indistinguishable from one another but clearly better than the alternatives. The study found that the differences between these top three models were so small that no single one could be declared the absolute winner; they are simply all excellent choices for this type of problem.

Perhaps the most surprising finding concerned the synthetic data. The researchers tested whether adding those fake, balanced data points improved the models. They found that it did not. Once the researchers adjusted the decision threshold—the specific point where the computer decides to say "yes" or "no"—based on a separate validation set, the need for synthetic data disappeared. In fact, for this type of moderately unbalanced financial data, simply tuning the decision point was enough to handle the imbalance. Creating fake data points was not only redundant but could potentially introduce noise that confused the model. This suggests that financial institutions can skip the complex step of generating synthetic data and instead focus on carefully calibrating their decision rules.

Beyond just predicting who will default, the study also looked at how well these models could explain their own decisions, a requirement for legal compliance. The researchers used two different methods to generate these explanations: one based on game theory and another based on local approximations. They found that while these two methods generally agreed on the most important factors, they did not always agree on every single person. For about sixty-two percent of the cases, the explanations were reasonably consistent, but for some individuals, the two methods pointed to different reasons for a denial. This disagreement is a critical risk. If a bank relies on only one method to tell a customer why they were rejected, and that method is unstable, the bank could face legal challenges. The study concludes that institutions should check for agreement between different explanation tools and have human experts review cases where the computer's reasoning is unclear.

The researchers also tested their findings on a smaller, older dataset from Germany to see if the results held up with less data. They found that with fewer examples, the advantage of the complex models over the simpler ones became harder to prove statistically, suggesting that the power of these advanced tools depends on having enough data to learn from. Finally, the team checked for fairness across different groups, such as men and women or people with different education levels. They found only small differences in how the model treated these groups, but they noted that for very small groups, the data was too sparse to draw firm conclusions. The study emphasizes that while the models performed well overall, they are trained on specific historical data and cannot be blindly applied to different populations or economic crises without re-evaluation.

This work establishes a new, cleaner way to test credit scoring models. It proves that the most effective tools are already available and that they do not need artificial data tricks to work well. More importantly, it highlights that accuracy is not enough; the explanations behind the decisions must be consistent and reliable. By following the strict, leakage-free protocol outlined in the study, financial institutions can build systems that are not only more accurate but also more trustworthy and legally sound. The code and data used in this research are now open for anyone to use, ensuring that this new standard can be adopted and verified by the wider community.