Technical Summary: Can Large Language Models Work as Post-hoc Explainability Tools in Credit Risk Models?

1. Problem Statement

The rapid adoption of complex machine learning (ML) models, such as XGBoost and neural networks, in FinTech lending has improved predictive accuracy but introduced opacity, posing challenges for regulatory compliance (e.g., ECOA, FCRA) and risk management. While post-hoc Explainable AI (XAI) methods like SHAP and LIME provide numerical feature attributions, they often lack the semantic clarity and narrative structure required for human decision-makers and regulatory reporting.

This study investigates whether Large Language Models (LLMs) can bridge this gap by serving as post-hoc explainability interfaces for credit risk models. Specifically, it addresses two distinct research questions:

• RQ1 (LLM as Translator): Can LLMs faithfully preserve and reproduce model-grounded feature-importance rankings when these rankings are explicitly provided as input?
• RQ2 (LLM as Autonomous Explainer): Can LLMs autonomously infer feature-importance rankings that align with model-grounded explanations (e.g., SHAP or logistic coefficients) without being provided the reference rankings, relying solely on instance data and prompting strategies?

2. Methodology

Data and Baseline Models

• Dataset: A publicly available LendingClub consumer loan dataset (2007–2011) comprising 39,239 loan applicants with 24 independent features. The target variable is a binary indicator of loan default (Charged Off vs. Fully Paid).
• Baseline Models: Two credit risk models were trained to generate ground-truth reference attributions:
  1. Logistic Regression: Provides inherently interpretable, coefficient-based linear attributions.
  2. XGBoost: A state-of-the-art tree-based ensemble model capturing non-linearities, with attributions derived via SHAP (SHapley Additive exPlanations).
• Evaluation Metrics: Predictive performance was assessed using PR-AUC, Macro-averaged F1-score, and the Kolmogorov–Smirnov (KS) statistic. XGBoost significantly outperformed Logistic Regression (PR-AUC: 0.38 vs. 0.27; KS: 40.4 vs. 28.79).

LLM Experimental Design

Three commercial LLMs were evaluated: GPT-4-turbo, Claude-Sonnet-4.5, and Gemini-2.5-Flash. Experiments utilized In-Context Learning (ICL) with temperature fixed at 0 to ensure reproducibility.

• RQ1 (Translator Setting):

  • Input: Instance features, predicted probability, and the reference ranked feature list derived from the baseline model.
  • Task: The LLM must output a ranked list of features strictly adhering to the provided reference order.
  • Constraint: Strict JSON output format; no external knowledge or inference allowed.
  • Metrics: Top-K Feature Overlap and Kendall's $\tau$ coefficient.

• RQ2 (Autonomous Explainer Setting):

  • Input: Instance features, predicted probability, and task instructions. No reference rankings provided.
  • Strategies: Compared Zero-shot (instructions only) vs. Few-shot (instructions + two demonstration examples of high/low risk cases with their reference rankings).
  • Conditions:
    1. Top-4: LLM returns only the top 4 most important features (simulating adverse action notices).
    2. Full-Ranking: LLM returns a complete ranking of all 24 features.
  • Metrics: Top-K Overlap and Kendall's $\tau$ against the ground-truth reference rankings.

3. Key Results

RQ1: LLM as Translator

• High Structural Fidelity: When reference rankings were explicitly provided, LLMs demonstrated near-perfect fidelity.
• Metrics: For all schema-valid outputs, Overlap@K was 1.000 and Kendall's $\tau$ was 1.000 across all cutoffs ($K=5, 10, 15, 20$) for both baseline models and all three LLMs.
• Robustness: Out of 3,000 total runs, only one instance of a feature-name substitution was observed (Claude-Sonnet-4.5 substituting "Past 2 Months" for "Past 2 Years"). No duplicate features or malformed JSON were found.
• Conclusion: LLMs are highly reliable at reformatting and reproducing supplied attribution structures under strict constraints.

RQ2: LLM as Autonomous Explainer

• Limited Alignment: Without reference rankings, LLMs showed only partial alignment with model-grounded explanations.
• Top-4 Condition:
  • Zero-shot: Mean Overlap@4 was very low (e.g., 0.064–0.199 for Logistic Regression; 0.102–0.233 for XGBoost).
  • Few-shot: Performance improved but remained insufficient for standalone use. For Logistic Regression, GPT-4-turbo improved to 0.515, but for XGBoost, improvements were marginal (max 0.383).
  • Observation: Few-shot prompting helped recover linear structures (Logistic Regression) better than complex, non-linear interactions (XGBoost).
• Full-Ranking Condition:
  • Schema Validity: Gemini-2.5-Flash frequently failed to generate complete 24-feature lists under long prompts, producing truncated outputs.
  • Alignment: Even for valid outputs (GPT-4-turbo, Claude-Sonnet), Overlap@K improved with few-shot prompting (e.g., Logistic Regression O@5 rose from ~0.39 to ~0.61) but never reached perfection.
  • Ordering Consistency: Kendall's $\tau$ values remained modest (mostly < 0.4), indicating that while LLMs might identify some relevant features, they fail to preserve the correct relative ordering of the full attribution structure.

4. Key Contributions and Claims

Primary Findings

The study establishes a clear distinction between LLMs as narrative interfaces versus autonomous explainers:

1. Reliability as Translators: LLMs can reliably reproduce model-grounded rankings when those rankings are provided as input. They function effectively as "narrative interfaces" that convert quantitative attributions into human-readable formats without altering the underlying logic.
2. Unreliability as Autonomous Explainers: LLMs cannot reliably infer feature importance rankings that match complex credit risk models (especially non-linear ones like XGBoost) solely from input data and prompts. Their outputs are driven by general domain priors and language patterns rather than the specific fitted model structure.

Significance for Credit Risk Governance

• Not a Substitute for Formal XAI: The paper argues that LLMs should not replace formal attribution methods (like SHAP or coefficient decomposition) in credit risk governance. Autonomous LLM explanations may select plausible factors that do not reflect the actual decision logic of the approved model, introducing model risk.
• Governance-Oriented Deployment: The recommended use case is a hybrid workflow: Formal XAI methods generate the auditable attribution rankings, and LLMs serve as the interface to translate these rankings into coherent, domain-informed narratives for stakeholders.
• Limitations: The study acknowledges limitations, including the use of a single dataset, the focus on rank alignment rather than user outcomes, and the lack of perturbation stress tests (e.g., shuffled rankings).

Conclusion

The paper concludes that while LLMs offer promise for enhancing the accessibility of credit risk explanations, they currently lack the fidelity to serve as independent sources of explanatory accountability. Their role should be constrained to formatting and narrating pre-computed, auditable attributions rather than inferring them autonomously.