All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models

Problem Statement

The paper addresses the persistent accuracy–explainability trade-off in Explainable AI (XAI). Conventional wisdom suggests that improving model interpretability (e.g., using intrinsically interpretable models) comes at the cost of predictive performance, while high-performing black-box models require post-hoc explanations that are often disconnected from the actual decision-making process.

The authors argue this trade-off is not fundamental but an artifact of treating explanation and prediction as separate objectives. They identify three specific limitations in existing XAI methods that prevent the realization of a coupled approach:

1. Distributional Fragility: No single explanation method remains valid under distribution shifts. Any simplification of a complex model will inevitably fail on some inputs, and a small shift toward these error regions exposes the failure (Theorem 1).
2. Inadequate Representation: Scalar attributions (e.g., SHAP, LIME) cannot effectively capture non-linear, conditional, or multivariate relationships, which are common in real-world data.
3. Decoupling: Most XAI methods operate independently of the prediction model. Explanations that do not contribute to prediction performance cannot be verified to reveal true data patterns, rendering them potentially just "stories" that fit observed data.

Methodology: RashomonLLM

To overcome these limitations, the authors propose the Rashomon Explanation paradigm and a specific implementation called RashomonLLM.

1. The Rashomon Explanation Paradigm
Instead of seeking a single "true" explanation, the authors define a Rashomon Explanation Set: a collection of explanations that are both faithful (close to the ground-truth feature-outcome relationships) and useful (capable of guiding a near-optimal prediction model). The final explanation is an aggregation of this set, leveraging "collective intelligence" to produce robust results.

2. The Explanation–Prediction–Reflection (EPR) Agentic Workflow
RashomonLLM utilizes Large Language Models (LLMs) within an iterative agentic framework consisting of three agents:

• Explanation Agent: Generates natural language explanations based on observed outcomes. To ensure diversity and avoid overfitting to a single pattern, it employs LLM Feature Dropout, randomly masking 20% of input features in each iteration to force the model to explore different feature subsets.
• Prediction Agent: Uses the generated explanations as context to predict outcomes. The explanations serve as instructions for the LLM to reason about the data.
• Reflection Agent: Analyzes prediction errors made by the Prediction Agent. It identifies which rules in the current explanation caused the errors and rewrites the explanation prompt to correct these specific failures without discarding working rules.

This loop continues until prediction error stabilizes, implementing double-loop learning (revising the framework itself) and complementarity (jointly optimizing explanation and prediction).

3. Scalability: Batch LLM Learning
To handle industrial-scale datasets that exceed LLM context windows, the authors introduce Batch LLM Learning. The training data is segmented into batches processed sequentially. A "Data Monitor" mechanism retains sentinel rows from previous batches to ensure the LLM maintains consistency and does not forget earlier data as the context grows.

Key Contributions

• Theoretical:
  • Theorem 1: Proves that any single explanation is sensitive to distribution shifts and will fail on an adversarial set.
  • Theorem 2: Establishes that the excess risk of an explanation-guided model is bounded by the explanation's fidelity; higher fidelity leads to better predictive performance.
  • Theorem 3: Proves the existence of a non-empty Rashomon Explanation Set under specific assumptions.
  • Theorem 4: Demonstrates that the EPR workflow converges geometrically and, with varied configurations (templates and dropout), can recover the full Rashomon Explanation Set.
• Methodological:
  • Introduces the RashomonLLM framework, the first method to couple explanation generation with prediction improvement via an iterative agentic loop.
  • Proposes LLM Feature Dropout and Batch LLM Learning as novel techniques to explore the explanation set and scale to large data.
• Empirical:
  • Validates the approach on two Kaggle benchmarks (Bank Churn classification, HCT Survival regression) and a large-scale industrial Click-Through Rate (CTR) prediction task using Kuaishou's live-streaming logs.

Experimental Results

The paper reports significant performance gains across all datasets:

• Prediction Accuracy: RashomonLLM outperformed state-of-the-art baselines, including specialized deep learning models (TabTransformer, DCNv2), winning Kaggle solutions, and strong classical models (Gradient-Boosted Trees). In the industrial CTR task, it achieved an accuracy of 0.771, surpassing the best baseline (DCNv2 at 0.761).
• Explanation Quality: The method produced explanations with significantly higher faithfulness (measured by Single Deletion and Randomization Check metrics) compared to post-hoc LLM explainers and sparse decision trees.
• Ablation Studies:
  • Removing the Reflection agent or the Iteration loop caused significant drops in both accuracy and explanation quality, confirming the necessity of the coupled EPR process.
  • Using a "placebo" explanation (random text) resulted in poor performance, proving that the gain comes from explanation fidelity, not merely the addition of text.
  • The performance gains were robust across different LLM backbones (Llama-3, Qwen3), temperatures, and feature orderings.
• Industrial Deployment: In the KuaiLive application, the model's accuracy gain translated to an estimated economic gain of ~$0.02 per 1,000 predictions, outweighing the serving costs.

Significance and Claims

The paper claims to fundamentally shift the understanding of the relationship between explainability and accuracy:

1. Complementarity, Not Trade-off: The authors demonstrate that equipping a model to explain itself improves its predictive accuracy. Explanation and prediction are shown to be complements rather than substitutes.
2. Plurality over Singularity: By embracing the "Rashomon" concept (multiple valid explanations), the system achieves robustness against distribution shifts that single-explanation methods cannot withstand.
3. Actionable Trust: The framework generates natural-language rationales that are directly consumable by non-technical stakeholders (e.g., product managers, regulators), bridging the gap between black-box performance and organizational accountability.
4. Boundary Conditions: The authors modestly note that the method is designed for structured, tabular data with semantically meaningful features. They explicitly state they do not claim to recover true causal mechanisms (Level 4 explanations) but rather population-level structures (Level 3) that are faithful to the data distribution.

In conclusion, the paper argues that by properly coupling explanation and prediction through an iterative, multi-agent workflow, organizations can simultaneously advance business performance and lay the groundwork for consumer trust.