🤖 machine learning

We Need Explanation Cards to Connect Explanation Algorithms to the Real World

This paper proposes "Explanation Cards" as a practical framework to bridge the gap between complex algorithmic explanations and real-world utility by augmenting standard outputs with robustness data, validity checks, and clear interpretation instructions that shift the responsibility of understanding from users to providers.

Original authors: Eric Günther, Balázs Szabados, Kristof Meding, Gunnar König, Sebastian Bordt, Ulrike von Luxburg

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eric Günther, Balázs Szabados, Kristof Meding, Gunnar König, Sebastian Bordt, Ulrike von Luxburg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a mysterious "Black Box" machine that makes important decisions for you, like whether to approve a loan or diagnose a medical condition. You ask the machine, "Why did you say no?" and it gives you a reason.

The problem, according to this paper, is that these reasons are often like bad maps. They look helpful, but if you try to follow them in the real world, you might get lost or end up somewhere dangerous. The people making the decisions (the banks, the doctors) know how the map was drawn, but the people receiving the decisions (the applicants, the patients) are left guessing if the map is accurate.

The authors propose a solution called Explanation Cards. Think of these as warranty labels or instruction manuals that must be attached to every explanation the machine gives. They don't just give you the answer; they tell you exactly how much you can trust that answer and what you can safely do with it.

Here is how the paper breaks this down using two main examples:

1. The Loan Applicant and the "Magic Button" (Counterfactual Explanations)

Imagine you apply for a loan and get rejected. The machine gives you a "Counterfactual Explanation." It's like a magic button that says: "If you had $14,000 in your savings instead of $4,000, you would have been approved."

The Trap:
You might think, "Great! I'll save that money, and I'll be approved." But you might also assume:

"If I save more than $14,000, I'll definitely be approved." (Monotonicity)
"This advice will still work even if I get a new job or my age changes slightly." (Stability)

The Reality:
The paper argues that for complex machines, the "decision boundary" (the line between yes and no) can be jagged and weird. Saving $15,000 might actually get you rejected because of some hidden rule about your age or job type. The machine's "magic button" might only work for a tiny, specific scenario. If you follow the advice blindly, you might fail.

The Solution (The Explanation Card):
The provider must attach a card that says:

"This advice works only if you keep your savings between $10,000 and $15,000."
"This advice stops working if you change your job or open a checking account."
"This advice is only for people in your specific age group."

This shifts the responsibility. Instead of you guessing if the advice is safe, the bank (the provider) must explicitly tell you the limits upfront.

2. The Doctor and the "Noisy Weather Forecast" (SHAP Explanations)

Now imagine a doctor uses an AI to diagnose heart disease. The AI gives a "SHAP value" for a patient's age, saying, "Age is a negative factor here."

The Trap:
The doctor might look at a chart and think:

"As patients get older, the risk goes up steadily." (Monotonicity)
"This chart shows exactly how age changes the risk." (Additivity)

The Reality:
The paper explains that in complex models, features interact like a tangled knot of wires. Changing a patient's age might accidentally change how the machine views their blood pressure or cholesterol. The chart might look like a smooth line, but the actual risk might be jumping up and down wildly behind the scenes. The chart is a "noisy forecast" that doesn't match the real weather.

The Solution (The Explanation Card):
The card acts like a confidence meter. It shows the doctor:

"Here is the range of patients we are looking at."
"Here is the 'envelope' of uncertainty."
If the envelope is small, the doctor can trust that the chart reflects reality.
If the envelope is huge (like a giant fog), the card explicitly warns: "Do not trust this chart to predict risk. The machine is too complex for this simple view."

The Big Picture: Why This Matters

The paper argues that currently, we expect users (like loan applicants or doctors) to be experts in math to understand these explanations. That's unfair.

Explanation Cards flip the script:

Before: "Here is a number. Good luck figuring out what it means."
After: "Here is the number, and here is a clear note saying exactly what you can and cannot conclude from it."

The authors also mention that this fits perfectly with the EU AI Act, a new law that requires companies to be transparent. Instead of just dumping raw data on users, companies can use these cards to legally and ethically explain their AI systems, ensuring that the "warranty" on the explanation is clear.

In short: Explanation Cards are the "Terms and Conditions" for AI explanations, written in plain language, so you don't get burned by a misleading map.

Technical Summary: Explanation Cards for Explanation Algorithms

1. Problem Statement

The paper addresses a critical gap in Explainable AI (XAI): while algorithmic explanations are intended to help stakeholders understand opaque decisions, they often fail in practice due to two primary issues:

Misinterpretation: The meaning of algorithmic explanations often diverges from intuitive human expectations. Stakeholders, particularly non-experts, frequently draw incorrect conclusions because they lack the expert knowledge required to interpret the technical output correctly.
Lack of Informativeness: Recent theoretical work (e.g., Günther et al., 2025) demonstrates that popular explanation algorithms (such as SHAP and counterfactuals) often provide no information about the behavior of complex decision functions. Specifically, these explanations may be unstable or rely on assumptions (like additivity or monotonicity) that the underlying model does not satisfy.

Consequently, there is a disconnect between what explanations appear to convey and what they actually provide, leading to a situation where explanations in high-stakes settings (e.g., loan approvals, medical diagnosis) may be misleading or useless.

2. Methodology: Explanation Cards

The authors propose Explanation Cards as a pragmatic solution to bridge the gap between technical explanation algorithms and real-world utility. An Explanation Card is a document that augments the standard output of an explanation algorithm with:

Complementary Information: Technical guarantees regarding the robustness and validity of the specific explanation (e.g., stability regions, monotonicity properties).
Interpretation Instructions: Clear, natural-language directives specifying what conclusions can and cannot be drawn from the explanation.

Core Philosophy: The responsibility for interpretation shifts from the user to the provider. Instead of expecting users to recognize the limitations of an explanation, providers must explicitly state the conditions under which the explanation is valid.

Technical Implementation

The paper derives mathematical guarantees to construct these cards for two specific algorithms:

A. Counterfactual Explanations

The Problem: Users intuitively assume counterfactuals imply monotonicity (increasing a feature further improves the outcome), stability of the counterfactual (the advice holds if other features change slightly), and stability around the input (similar individuals receive similar advice). However, for complex models (e.g., random forests), decision boundaries can be non-monotonic and unstable, rendering these intuitive interpretations invalid.
The Card Solution: The card provides:
- Stability Regions: Explicitly defines the "region of stability" (a hyper-rectangle) where the counterfactual remains valid.
- Monotonicity Checks: States whether the counterfactual is monotonic in specific features.
- Local Validity: Identifies the specific demographic or feature range (e.g., "applicants aged 30-35 with income $45k-$ 55k") for whom the advice applies.
Theoretical Basis: The authors prove that counterfactuals lacking these stability regions are "non-informative" (Theorem 1), whereas those with stable regions are "informative" (Theorem 2).

B. SHAP (SHapley Additive exPlanations)

The Problem: Users often interpret SHAP dependence plots as reflecting the true marginal effect of a feature on the prediction (implying additivity). However, SHAP values depend on all features; if the model has strong interactions, a SHAP dependence plot may show a trend that contradicts the actual change in prediction when the feature is varied.
The Card Solution: The card includes:
- Local Region Definition: Specifies the neighborhood (e.g., 50 nearest neighbors) used for the analysis.
- Prediction Bounds: Instead of just showing the SHAP value trend, the card plots the maximum and minimum change in prediction (dotted envelope) when the feature is varied while holding other features constant within the local region.
- Validity Assessment: If the "envelope" is narrow, the SHAP plot roughly reflects the prediction behavior (approximate additivity). If the envelope is wide, the plot is misleading, and the card explicitly warns the user against interpreting the trend as a causal or predictive effect.
Theoretical Basis: The paper defines "local approximate invariance" of SHAP values. It proves that if SHAP values are approximately invariant to changes in other features within a region, the model is approximately additive in that region (Theorem 6), making the dependence plot interpretable.

3. Key Contributions

Conceptual Framework: The proposal of Explanation Cards as a mechanism to operationalize the "informativeness" of explanations by translating technical guarantees (stability, monotonicity, additivity) into user-facing instructions.
Mathematical Derivation: The derivation of specific mathematical conditions (Theorems 1, 2, 4, 6) that determine when counterfactuals and SHAP values are informative, and how to construct cards based on these conditions.
Concrete Examples: The construction of two detailed explanation cards:
- A card for counterfactuals in loan approval, demonstrating how to communicate stability regions and monotonicity constraints.
- A card for SHAP in medical diagnosis, demonstrating how to visualize prediction bounds to validate (or invalidate) intuitive interpretations of feature importance.
Legal Analysis: An analysis of how Explanation Cards can help providers of high-risk AI systems comply with Article 13 of the EU AI Act, which mandates transparency and instructions for deployers to interpret system outputs.

4. Results and Demonstrations

The paper does not present a large-scale empirical study but rather a theoretical and illustrative demonstration:

Counterfactuals: Using a decision tree and a random forest on credit data, the authors show that while a decision tree yields a stable, monotonic counterfactual (valid for intuitive interpretation), a random forest yields a counterfactual that is unstable and non-monotonic. The Explanation Card successfully distinguishes these cases, warning the user in the latter scenario.
SHAP: Using the UCI Heart Disease dataset, the authors demonstrate that for a random forest model with interactions, the SHAP dependence plot for "age" suggests an increasing risk, while the actual prediction change (measured by the dotted envelope) shows a decreasing risk. The Explanation Card reveals this discrepancy via a wide envelope, preventing misinterpretation. Conversely, for an additive model, the envelope is narrow, validating the intuitive interpretation.

5. Significance and Claims

The authors position Explanation Cards as a significant step toward making explanation algorithms "fit for real-world use cases."

Operationalizing Explainability: The paper argues that Explanation Cards provide a practical means to satisfy the transparency and explainability provisions of the EU AI Act (specifically Art. 13), shifting the burden of ensuring interpretability from the end-user to the system provider.
Preventing Harm: By explicitly stating what cannot be concluded from an explanation, the cards aim to prevent stakeholders from making dangerous decisions based on misleading intuition.
Modest Scope: The authors acknowledge that Explanation Cards are not a universal fix. They do not claim to make every explanation meaningful; rather, they provide a framework to identify when an explanation is meaningful and to communicate those limits clearly. The paper serves as a "first step," calling for further research to derive similar guarantees for other algorithms and scenarios.