From Model Explanation to Data Misinterpretation: A Cautionary Analysis of Post Hoc Explainers in Business Research

This paper cautions against treating post hoc explainers like SHAP and LIME as definitive evidence for underlying data relationships in business research, demonstrating through a systematic review and simulation that their explanations often misalign with true data-generating processes due to feature correlation and the Rashomon effect, and thus should be used only as exploratory tools rather than for hypothesis validation.

The Gist

Here is an explanation of the paper, translated from academic jargon into a story about detectives, maps, and a very tricky game of "Guess the Rule."

The Big Problem: The "Black Box" Detective

Imagine you hire a super-smart detective (a Machine Learning Model) to solve a mystery. Your detective is amazing at predicting the outcome: "Will this customer buy the product?" or "Will this loan default?" They get it right 95% of the time.

But there's a catch: The detective is a Black Box. They give you the answer, but they won't tell you why. They just say, "I know it's true because I know it."

Business researchers are desperate to know why. They want to know: "Does age matter? Does income matter? Is it the weather?"

So, they hire a second detective, called a Post-Hoc Explainer (like SHAP or LIME). This second detective's job is to look at the first detective's work and say, "Ah, I think the first detective was looking at Income to make that decision."

The Mistake:
The paper argues that researchers are making a huge error. They are taking the second detective's guess and treating it as absolute truth about how the world works. They say, "The second detective says Income matters, therefore Income definitely causes people to buy things."

The authors say: Stop! The second detective is just guessing what the first detective was thinking. They aren't necessarily telling you the truth about the real world.

---

The Analogy: The "Many Paths" Mountain

To understand why this is dangerous, imagine a mountain with a peak (the correct answer).

1. The Rashomon Effect (The Many Paths):
   Imagine there are 100 different paths up the mountain. All 100 paths lead to the exact same peak. You can't tell which path is the "real" one just by looking at the top; they all look perfect.

   • Path A goes through the forest.
   • Path B goes along the river.
   • Path C goes over the rocks.
   • The Result: All paths get you to the top (high accuracy). But the scenery (the features used) is totally different.

2. The Explainer's Job:
   You ask a guide (the Explainer) to describe the path.

   • If you ask about Path A, the guide says, "We walked through the forest!"
   • If you ask about Path B, the guide says, "We walked by the river!"
   • Both guides are telling the truth about the path they were on. But neither guide is telling you the only way to the top.

3. The Danger:
   Researchers pick Path A, ask the guide, and then conclude: "The only way to the top is through the forest!" They ignore the fact that Path B and Path C also work. They mistake the path taken by one specific model for the laws of nature.

What the Authors Did (The Experiment)

The authors (Tong Wang, Ronilo Ragodos, Lu Feng, and Yu Jeffrey Hu) decided to test this. They created a "Ground Truth" simulation.

• The Setup: They built a fake world where they knew the exact rules (e.g., "Income always increases the chance of buying, but only if Age is over 30").
• The Test: They let AI models learn these rules. Then, they asked SHAP and LIME to explain what the models learned.
• The Result:
  • On Average: The explainers were okay. They usually got the general idea right.
  • In Reality (The Long Tail): When they looked closely, the explainers were often wrong. Sometimes they said "Age" was the most important factor when it wasn't. Sometimes they said "Income" had a positive effect when it actually had a negative one.
  • The Shock: Even when the AI model was 99% accurate at predicting the outcome, the explanation it gave was sometimes completely flipped or wrong. High accuracy does not guarantee a truthful explanation.

Why Does This Happen?

The authors found three main reasons why the "guide" gets confused:

1. The Rashomon Effect (Too many paths): As mentioned, if many different models can predict the outcome equally well, they might all use different "clues." One model might rely on "Income," another on "Education." If you pick one model at random, its explanation is just one random guess among many.
2. Correlated Features (The Twin Problem): Imagine "Income" and "Education" are twins; they always go up and down together. The AI can't tell which one is actually doing the work. It might pick "Income" for one model and "Education" for another. The explainer will point to one, but it's just a coin flip.
3. Complexity: If the real world is messy (non-linear, with lots of interactions), it's harder for the AI to find the one true rule, so it finds a "good enough" rule that looks right but isn't the real story.

The Solution: How to Check if You're Being Lied To

The authors don't say "Stop using AI." They say, "Stop trusting the AI's explanation as the final truth."

Instead, they propose a Reliability Check:

• The "Group Consensus" Test:
  Don't just ask one detective. Ask 10 different detectives who all got the same score on the test.
  • Scenario A: All 10 detectives say, "We all looked at Income." -> Good! You can trust this. The "Rashomon Agreement" is high.
  • Scenario B: 5 detectives say "Income," 3 say "Education," and 2 say "Age." -> Bad! The "Rashomon Agreement" is low. This means the data is ambiguous. You cannot trust any single explanation.

The Bottom Line: What Should Business Researchers Do?

The paper gives a simple rule of thumb:

• Don't use Explainers for Validation: Do not use SHAP or LIME to prove a hypothesis (e.g., "We proved that X causes Y"). They are not rigorous enough for that.
• Do use Explainers for Exploration: Use them to generate ideas. "Hey, the AI thinks Income is important. That's an interesting idea! Let's go run a proper scientific experiment to see if that's actually true."

In short: Treat Post-Hoc explainers like a weather forecast, not a law of physics. A forecast might be right 90% of the time, but if you bet your life on it, you might get wet. Use them to get hints, but always double-check with real-world evidence.

Technical Summary

Here is a detailed technical summary of the paper "From Model Explanation to Data Misinterpretation: A Cautionary Analysis of Post Hoc Explainers in Business Research."

1. Problem Statement

The paper addresses a critical methodological flaw in contemporary business research: the misinterpretation of post hoc explainers (specifically SHAP and LIME).

• The Trend: Researchers increasingly use post hoc explainers not just to interpret model behavior ($X \to \hat{Y}$), but as evidence to infer the true underlying data-generating process ($X \to Y$). They treat feature attributions (direction and strength) as direct indicators of causal or associative relationships in the data.
• The Gap: While these tools are designed to explain the model, there is no rigorous validation that they accurately recover the true data relationships. The authors argue that high predictive accuracy does not guarantee that an explainer's output reflects the ground truth, leading to potential "data misinterpretation."

2. Methodology

The authors employ a multi-stage approach combining a systematic literature review, controlled simulation experiments, and diagnostic analysis.

A. Systematic Literature Review

• Sample: 181 studies from top-tier business journals (UTD 24, FT50, INFORMS) and preprints.
• Coding: They identified two primary types of interpretations:
  1. Direction Interpretation: Inferring whether a feature increases or decreases the outcome based on the sign of the attribution.
  2. Strength Interpretation: Inferring the relative importance of features based on the magnitude of attributions.
• Finding: Approximately 42.5% of the reviewed papers incorrectly generalized model-level explanations to data-level inferences.

B. Simulation Framework & Metrics

To test validity, the authors generated 81 simulated datasets with known ground truths ($G$) where the relationship between features ($X$) and outcomes ($Y$) was precisely controlled.

• Variables: They systematically varied feature count, correlation strength, nonlinearity, and interaction terms.
• Models: They trained various ML models (XGBoost, Random Forest, etc.) to achieve high predictive accuracy.
• Explainers: Applied SHAP and LIME to these models.
• New Metrics:
  1. Direction Alignment ($\rho_{dir}$): Measures if the sign of the explainer's implied change matches the true change in $Y$ when a feature is perturbed.
  2. Strength Alignment ($\rho_{strength}$): Measures if the ranking of feature importance (based on mean absolute attributions) matches the true ranking of importance in the ground truth (using Spearman correlation).

C. Diagnostic Analysis (Rashomon Sets)

The authors investigated the Rashomon Effect—the existence of many distinct models with near-identical predictive performance but different internal logic.

• They constructed "Rashomon sets" (groups of top-performing models) for each dataset.
• They defined Rashomon Agreement:
  • Prediction Agreement: Frequency of identical predictions between models.
  • Explanation Agreement: Correlation of feature importance rankings between models.

3. Key Contributions

1. Conceptual Distinction: The paper formally distinguishes between Model Explanation (validating $X \to \hat{Y}$) and Data Inference (inferring $X \to Y$), arguing that post hoc explainers are valid for the former but unreliable for the latter.
2. Quantification of Misalignment: It provides the first systematic evidence that even with high predictive accuracy, post hoc explanations frequently fail to align with ground truth direction and strength.
3. Identification of Drivers: It isolates feature correlation and the Rashomon Effect as the primary drivers of misalignment, rather than just model complexity or explainer choice.
4. Proposed Diagnostic: It introduces Rashomon Agreement (specifically explanation agreement) as a practical, dataset-level diagnostic tool to assess the reliability of an explanation before trusting it.

4. Key Results

A. Performance vs. Reliability

• High Average, High Variance: On average, SHAP and LIME show high alignment with ground truth. However, the distributions exhibit pronounced long left tails.
• The "Black Swan" of Explanations: A non-trivial subset of dataset-model pairs shows severe misalignment (e.g., strength alignment dropping to ~0.5, implying random ranking), even when the model's predictive accuracy is high (>85%).
• Necessary but Insufficient: High predictive accuracy is necessary for alignment but insufficient to guarantee it.

B. Drivers of Misalignment

• Feature Correlation: This is the dominant driver. High correlation allows models to substitute features, leading to different internal attributions for the same predictive performance.
• Rashomon Effect: Models with nearly identical accuracy can rely on completely different feature subsets or interaction structures. Consequently, their explanations can be contradictory.
• Data Complexity: Nonlinearities and interaction terms increase the size of the Rashomon set, further reducing the reliability of single-model explanations.

C. The Diagnostic Power of Agreement

• Explanation Agreement is Key: There is a strong positive correlation between Explanation Agreement (consistency of feature rankings across a Rashomon set) and Alignment with ground truth (Spearman $\rho \approx 0.79$ for SHAP).
• Prediction Agreement is Weak: Consistency in predictions between models does not correlate well with alignment (Spearman $\rho \approx 0.09$). Models can agree on the output but disagree on why (the features used).
• Conclusion: If equally accurate models disagree on feature importance, the explanation is likely unreliable.

5. Significance and Implications

For Business Research

• Paradigm Shift: The paper argues that post hoc explainers should not be used for hypothesis validation or as evidence for data-level relationships (e.g., "Feature X causes Outcome Y").
• New Role: They should be repositioned as exploratory tools for hypothesis generation. They can suggest candidate variables or mechanisms, but these hypotheses must be validated using rigorous identification strategies (e.g., causal inference, randomized experiments, or structural econometrics).
• Risk Mitigation: Researchers should calculate Rashomon Agreement (specifically explanation agreement) before drawing conclusions. Low agreement serves as a red flag that the data structure is not identifiable enough to support specific claims about feature importance.

For the Field of XAI

• The findings highlight a fundamental limitation in current XAI practices: the assumption that a single model's explanation is a proxy for the truth.
• It underscores the need for methods that explicitly address the Rashomon effect (e.g., aggregating explanations across model sets) rather than relying on single-model post hoc analysis.

Summary Conclusion

The paper demonstrates that treating SHAP/LIME outputs as ground truth is a widespread and dangerous practice in business research. Due to the Rashomon effect and feature correlations, high-performing models can yield contradictory explanations. The authors conclude that predictive accuracy does not equal explanatory truth, and researchers must adopt a diagnostic approach (checking for model agreement) and a conservative workflow (using explainers for exploration, not validation) to avoid data misinterpretation.