Evaluating Local Explainability Metrics for Machine Learning Models on Tabular Data
This paper evaluates the reliability of local explainability metrics (LIME, SHAP, and Feature Ablation) across 32 tabular datasets and various models, revealing that explanation quality depends more on dataset complexity and feature distributions than on the model's predictive performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very smart, but mysterious, "black box" machine that makes decisions about things like loan approvals or medical diagnoses. You know it works well, but you don't know why it made a specific choice. To fix this, we use "explanation tools" (like LIME, SHAP, and Feature Ablation) that act like a translator, trying to tell us, "The machine said 'No' because your income was low."
But here's the problem: Just because the translator sounds convincing doesn't mean it's telling the truth.
This paper is like a quality control inspector for these translators. The authors asked: When we ask these tools to explain a decision on complex data (like a spreadsheet full of numbers and categories), are they actually being honest about how the machine thinks, or are they just making things up that sound good?
Here is a breakdown of what they did and what they found, using simple analogies:
1. The Setup: The "Taste Test"
The researchers gathered 32 different datasets (think of these as 32 different recipe books with thousands of ingredients). They trained three different types of "chefs" (machine learning models) to cook up predictions:
- Logistic Regression: A simple, straightforward chef who follows a basic recipe.
- Random Forest & XGBoost: Complex, super-chefs who use many different techniques and can handle very complicated recipes.
They then asked the three "translators" (LIME, SHAP, and Feature Ablation) to explain the chefs' decisions.
2. The Three Tests
To see if the translators were reliable, they ran three specific tests:
- Faithfulness (The "Truth Test"): Does the explanation match the chef's actual logic?
- Analogy: If the translator says, "The chef rejected the soup because of the salt," does the soup actually taste bad because of the salt? Or is the translator just guessing?
- Robustness (The "Stability Test"): If you wiggle the ingredients slightly (like adding a tiny pinch more salt), does the explanation stay the same, or does it flip completely?
- Analogy: If you change the recipe by a tiny bit, does the translator suddenly say, "Oh wait, it was actually the pepper!"? A good translator shouldn't flip-flop on tiny changes.
- Complexity (The "Simplicity Test"): How many ingredients does the translator blame for the decision?
- Analogy: A good explanation is like a short sentence: "It was the salt." A bad, complex explanation is a novel: "It was the salt, the pepper, the temperature, the time of day, and the color of the bowl."
3. The Big Surprise: "Right Answer" Doesn't Mean "Good Explanation"
The researchers looked at two groups of predictions:
- Consensus-Correct: When all the chefs got the answer right.
- Consensus-Wrong: When all the chefs got the answer wrong.
The Finding: They expected that when the chefs got it right, the translators would also be honest. They were wrong.
The quality of the explanation had very little to do with whether the machine got the answer right or wrong. Even when the machine was making a mistake, the translator could still give a very "stable" and "plausible" explanation.
4. The Real Culprit: The "Messy Kitchen"
The study found that the reliability of the explanation depended less on the machine's performance and more on how messy the data was.
- Analogy: Imagine trying to explain a recipe. If the kitchen has 5 ingredients, it's easy to explain. If the kitchen has 1,700 ingredients (a very complex dataset), it becomes incredibly hard for the translator to figure out which specific ingredient caused the result.
- The Result: The more complex the dataset (more columns, more features), the harder it was for the translators to be faithful and stable. The "messiness" of the data confused the translators, regardless of how smart the machine was.
5. How the Translators Performed
- LIME: It was the most stable (it didn't flip-flop much when ingredients were tweaked), but it was often unfaithful (it didn't always tell the truth about what the machine was thinking). It's like a calm liar.
- Kernel SHAP: It was often faithful (told the truth), but it was unstable. If you tweaked the data slightly, its explanation could go wild. It's like a truth-teller who gets very nervous and changes their story if you look at them sideways.
- Feature Ablation: This method (which tests what happens when you remove an ingredient) was generally the most sparse (it gave simpler explanations), but it could have extreme "worst-case" spikes where it became very unstable.
The Bottom Line
The paper concludes that we cannot trust an explanation just because the machine is accurate.
If you have a complex spreadsheet with hundreds of columns, the "translators" we use today might give you a story that sounds reasonable and stays consistent, but it might not actually reflect how the machine made its decision. The complexity of the data itself is the biggest factor in whether these explanations are trustworthy, not the skill of the machine making the prediction.
In short: A smart machine making a correct prediction doesn't guarantee a truthful explanation. If the data is too complex, the explanation tools might just be guessing, even if they sound convincing.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.