💻 computer science

SHAP and LIME Explainability in Financial Machine Learning: A Systematic Review of Methods, Evaluation Gaps, and Barriers to Deployment

This systematic review of 297 studies reveals that while SHAP and LIME are widely adopted for explainability in financial machine learning—particularly for credit modeling—they suffer from a critical lack of formal evaluation, with only 9.1% of papers assessing explanation quality, highlighting an urgent need for stronger validation and deployment standards to ensure regulatory compliance and trust.

Original authors: Manpreet Singh, Akshatha Srikantha, Keshav Kaushik, Prajwal Bajpai

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manpreet Singh, Akshatha Srikantha, Keshav Kaushik, Prajwal Bajpai

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern financial world, banks, insurers, and trading firms rely heavily on complex computer programs to make high-stakes decisions. These programs, often called machine learning models, can predict whether a person will repay a loan, flag a transaction as fraudulent, or determine the price of an insurance policy. While these tools are incredibly powerful, they often operate as "black boxes." This means that even the people who built them cannot easily see how the computer arrived at a specific answer. When a loan is denied or an account is frozen, the customer, the auditor, and the regulator need to know why. To solve this, researchers have developed a field called Explainable AI. These are tools designed to shine a light on the black box, translating the computer's complex math into human-readable reasons. Two of the most popular tools for this job are known as SHAP and LIME. They act like translators, taking a finished prediction and working backward to show which factors, such as income or spending history, mattered most. However, just because a tool can produce an explanation does not mean that explanation is correct, stable, or trustworthy enough to be used in a real courtroom or bank branch.

A team of researchers from Boston University, the University of California, Irvine, Sharda University, and the Indian Institute of Technology Kanpur decided to investigate exactly how well these tools are being used in the real world of finance. They did not just look at new ideas; they conducted a massive, systematic review of nearly 300 scientific studies published between 2016 and 2026. Their goal was to audit the evidence. They wanted to see if researchers were merely showing off pretty pictures of how a model works, or if they were actually proving that those pictures were accurate and reliable. The team followed a strict protocol, searching through thousands of academic records to find only those studies where SHAP or LIME were genuinely applied to financial data, such as credit scoring or fraud detection. They then examined every detail: what kind of financial problem was being solved, what kind of computer model was being explained, and most importantly, whether the researchers tested the quality of their explanations.

The results of this review reveal a significant gap between how fast these tools are being adopted and how carefully they are being tested. The researchers found that the field has moved very quickly. The vast majority of the studies they analyzed, about 74 percent, focused on simple yes-or-no decisions, like approving a loan or spotting fraud. In these studies, the tool known as SHAP was the clear favorite, appearing in nearly 94 percent of all the papers. This popularity is largely because SHAP works very well with the specific type of computer models that banks use most often. The second tool, LIME, was used much less frequently and usually only as a side note to SHAP, rather than as a primary method. While the researchers found that almost every study reported how well their computer model predicted the outcome, the story changed completely when they looked at the explanations themselves.

The most striking finding was that almost no one was actually checking if their explanations were any good. Out of the 297 studies reviewed, only 21, which is roughly 7 percent, conducted a formal test to see if the explanation matched the model's true behavior. Another 6 studies offered some limited evidence, but the overwhelming majority simply generated an explanation and assumed it was correct. The researchers noted that this is a dangerous oversight. In finance, an explanation is not just a visual aid; it is a critical part of the decision-making process. If a bank tells a customer they were denied a loan because of their credit history, but the explanation is actually wrong or unstable, it creates legal and ethical risks. The review found that researchers often treated the act of generating an explanation as the same thing as validating it, but the evidence shows these are two very different things.

The study also looked at the hurdles that prevent these tools from being used safely in real-world systems. The researchers found that 72 percent of the studies mentioned technical or data-related barriers. The most common problem cited was "class imbalance," a situation where the event being predicted, like a loan default or a fraud case, happens very rarely compared to normal cases. This rarity makes it difficult for the explanation tools to give consistent answers. Other barriers included the need for fast results, privacy concerns, and the fact that financial data changes over time. However, the review highlighted a disconnect: while researchers frequently listed these barriers as problems, they rarely tested how these issues actually damaged the quality of the explanations. They named the obstacles but did not measure the damage.

When the researchers looked at how scientists tried to build confidence in their results, they found that most relied on weak methods. About 36 percent of the studies mentioned some form of validation, but the most common approach was simply comparing the results of SHAP with LIME. If the two tools agreed, researchers often assumed the explanation was correct. The review argues that this is not enough proof, because both tools could be wrong in the same way. Stronger methods, such as testing how the explanation holds up when data changes slightly, or asking human experts to judge the results, were used in only a tiny fraction of the studies. The authors concluded that the field has mastered the art of producing explanations but has largely failed to prove that those explanations are trustworthy. They propose a new standard for future research, urging scientists to stop treating the generation of an explanation as the final step. Instead, they must treat the explanation itself as a product that needs rigorous testing, just like the financial model it describes, before it is ever used to make decisions that affect people's lives.

Technical Summary: SHAP and LIME Explainability in Financial Machine Learning

Problem Statement
Financial institutions increasingly rely on complex, high-performing machine learning (ML) models for critical decisions such as credit scoring, fraud detection, and risk assessment. While these models offer superior predictive performance, their "black-box" nature creates significant challenges regarding trust, fairness, accountability, and regulatory compliance. Explainable AI (XAI) is proposed as a solution to clarify model behavior. However, the current literature exhibits a critical fragmentation: while the application of post-hoc explanation methods like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) is expanding rapidly, there is a severe lack of rigorous evidence validating the quality, stability, and deployability of these explanations. The field risks conflating the mere generation of an explanation with the proof of its reliability, creating a gap between academic demonstration and operational viability in regulated financial environments.

Methodology
This study conducts a Systematic Literature Review (SLR) following PRISMA 2020 guidelines and software engineering protocols.

Scope: The review focuses exclusively on empirical studies where SHAP, LIME, or both are genuinely applied as post-hoc explanation methods to trained financial ML models.
Data Sources: Searches were conducted in May 2026 across Scopus, Web of Science, and IEEE Xplore.
Selection Process: From an initial pool of 4,168 records, 3,010 unique records were screened. After full-text assessment, 297 peer-reviewed studies met the inclusion criteria. A secondary "high-coverage" subset of 97 studies (those reporting evidence for at least three of the four research dimensions) was used for sensitivity analysis.
Analytical Framework: The review addresses four Research Questions (RQs):
1. RQ1: Mapping of XAI techniques to financial tasks and model families.
2. RQ2: Explicit metrics and methods used to evaluate explanation quality.
3. RQ3: Technical and data-related barriers to deployment.
4. RQ4: Broader methodological strategies used to validate explanations beyond single attribution outputs.
Coding: Data extraction was performed by two independent reviewers with high inter-coder reliability (Cohen's $\kappa \ge 0.78$ ). The analysis distinguishes between "explanation generation" (applying the method) and "explanation evaluation" (testing the quality of the output).

Key Contributions
The paper offers four primary contributions to the field of financial machine learning:

Technique-Task Mapping: A comprehensive mapping of SHAP and LIME applications to specific financial domains (e.g., credit, fraud, markets) and ML tasks (e.g., classification, time-series).
Evaluation Audit: A systematic documentation of how researchers evaluate explanations, tracking everything from fidelity metrics to user studies, and explicitly noting the prevalence of studies that skip quality assessment entirely.
Barrier Analysis: An identification of technical and data-related obstacles (e.g., class imbalance, latency, privacy) that prevent the transition from research prototypes to production systems.
Validation Strategy Categorization: A classification of validation approaches, distinguishing between low-assurance methods (e.g., simple method agreement) and high-assurance strategies (e.g., causal reasoning, adversarial testing, human-in-the-loop evaluation).

Key Results

Dominance of SHAP and Classification: The corpus is heavily skewed toward SHAP (present in 93.9% of studies) and classification tasks (73.7% of studies). This is largely driven by the native integration of TreeSHAP into gradient-boosted tree frameworks (XGBoost, LightGBM, CatBoost), which dominate tabular financial prediction. LIME is predominantly used as a secondary or corroborating method rather than a standalone tool.
Critical Evaluation Gap: Only 27 studies (9.1%) report any evidence of explanation quality, and only 21 studies (7.1%) conduct formal evaluations (e.g., fidelity, stability, robustness tests). The vast majority of studies generate visual explanations without validating their faithfulness to the underlying model.
Deployment Barriers: 72.4% of studies report at least one deployment barrier. Class imbalance is the most frequently cited barrier (117 studies), yet few studies empirically test how imbalance degrades explanation stability. Other barriers include data quality, latency, and concept drift.
Weak Validation Strategies: While 35.7% of studies report some validation practice, the most common strategies are low-assurance, such as "method agreement" (comparing SHAP and LIME outputs). Stronger strategies like causal analysis, counterfactual testing, and human-in-the-loop evaluation are rare. No study in the corpus reported a full "application-grounded" evaluation involving real users making live financial decisions.
Dataset Limitations: 81% of studies rely on public datasets, limiting the external validity of findings regarding real-world operational complexities like temporal drift and delayed labels.

Significance and Claims
The paper claims that the field of financial XAI has successfully achieved widespread adoption of explanation tools but has failed to demonstrate their reliability. The authors argue that the current literature treats the presence of an explanation method as equivalent to trustworthy explainability, which is a dangerous conflation for high-stakes financial decisions.

The significance of this review lies in its "evidence audit" approach. It moves beyond cataloging methods to rigorously assessing whether the generated explanations are actually valid. The authors conclude that:

Validation is the weakest link: The field must shift from generating explanations to validating them against specific financial tasks, stakeholders, and regulatory constraints.
Method agreement is insufficient: Relying on the agreement between SHAP and LIME does not prove faithfulness; stronger validation designs (causal, adversarial, human-centered) are necessary but underutilized.
Minimum Reporting Standards: The authors propose a new minimum reporting standard (Table 14) requiring researchers to define decision contexts, intended users, and at least one specific explanation-quality test. This is intended to force the evidence base to become cumulative and transparent, distinguishing between exploratory analyses and deployable systems.

Ultimately, the paper asserts that for XAI to be viable in regulated finance, the explanation itself must be treated as a "first-class object" subject to the same rigorous validation, monitoring, and governance as the predictive model itself.

Technical Summary: SHAP and LIME Explainability in Financial Machine Learning

More like this