🤖 machine learning

Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

The paper argues that while reliability and faithfulness are necessary conditions for interpreting scientific machine learning models, they are insufficient on their own to validate claims about the true structural mechanisms of the underlying phenomena without external corroboration.

Original authors: Nick Oh, Helen Jin

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nick Oh, Helen Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Black Box" Detective

Imagine scientists are trying to understand a complex natural phenomenon (like how a disease works or how a star burns). They build a super-smart computer model (an AI) to predict what happens. The model is incredibly accurate—it gets the right answer almost every time. This is Reliability.

But the model is a "black box." We don't know how it got those answers. So, scientists use special tools (like SHAP or LIME) to peek inside and ask, "What clues did you use to make that decision?" The tool gives an answer, like "The model looked at the patient's age and the color of their skin." This is Faithfulness.

The Paper's Main Argument:
The authors argue that even if the model is perfectly reliable (always right) and the explanation is perfectly faithful (tells the truth about what the model did), you still cannot conclude that the model actually understands how the real world works.

You can know what the machine did, but you don't know if it did it for the right reasons.

The Three-Step Chain

The paper describes a chain of logic that scientists often try to use, which looks like this:

The Real World (The Phenomenon): The actual thing we are studying (e.g., a real disease).
The Model: The AI that predicts the disease.
The Explanation: The tool that tells us what the AI was thinking.

The Mistake: Scientists often assume:

"The Model matches the Real World (Reliability)."
"The Explanation matches the Model (Faithfulness)."
Therefore: "The Explanation matches the Real World."

The Paper says: This logic is broken. The chain is missing a crucial link.

Analogy 1: The Cheating Student vs. The Genius

Imagine a student taking a math test.

Reliability: The student gets every single answer correct. They are a perfect predictor of the right answers.
Faithfulness: A proctor watches the student and reports exactly what the student did. The report says, "The student solved the problems by looking at the last digit of the question number and guessing the answer based on a pattern in the test booklet."

The Trap:
If you only look at the Reliability (perfect score) and the Faithfulness (the proctor's honest report), you might think, "Wow, the student has a brilliant new method for solving math!"

But in reality, the student is just cheating using a trick that works only on this specific test. They didn't learn the math (the structure of the phenomenon); they just found a shortcut that happens to work.

The paper argues that AI models often do this. They find "shortcuts" (like looking at the background of a photo instead of the disease) that make them accurate, but those shortcuts aren't how the real world actually works.

Analogy 2: The Two Clocks

Imagine you have two clocks.

Clock A is a real, high-quality clock with gears that match the movement of the sun.
Clock B is a broken clock that happens to be stuck at the exact time the sun is at, but only because someone set it that way.

Both clocks show the correct time (Reliability).
If you take a photo of the inside of both clocks, you get a faithful description of their insides (Faithfulness).

Clock A's inside shows gears turning.
Clock B's inside shows a frozen hand.

If you only know that both clocks show the right time, and you have a faithful description of their insides, you still cannot know which one is actually tracking the sun correctly. One of them is just lucky.

The paper says that Reliability and Faithfulness are like checking if the clocks show the right time and describing their insides. Neither check tells you if the clock is actually connected to the sun (the real mechanism).

The "Perfect" Scenario Doesn't Help

You might think, "What if the model is perfect? What if it never makes a mistake?"

The paper says: Even then, it doesn't matter.

Imagine a model that predicts a disease perfectly 100% of the time.

Scenario 1: The model learned the real biological cause (the "right" structure).
Scenario 2: The model learned a weird, invisible pattern in the data (like the hospital where the patient was treated) that happens to correlate with the disease, but isn't the cause.

Both models are 100% Reliable.
Both models have 100% Faithful explanations (telling you exactly what they looked at).

But in Scenario 2, the explanation is lying to you about the cause of the disease, even though it is telling the truth about the model. The model is right about the result, but wrong about the reason.

The Conclusion: What Can We Actually Do?

The paper concludes that we cannot use these explanations to make strong claims about how the world actually works (mechanisms) just by looking at the model and its explanation.

What the chain CAN do: It can give us ideas or hypotheses. It can say, "Hey, the model thinks this feature is important. Maybe we should go investigate that feature in the real world."
What the chain CANNOT do: It cannot prove that the feature is actually the cause.

To know the truth, scientists need external help. They need to bring in other knowledge, run real-world experiments, or use existing theories to verify if the model's "shortcut" is actually a real mechanism. The model and its explanation alone are not enough to bridge the gap between "the computer is right" and "we understand the universe."

Technical Summary: Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

Problem Statement
The paper addresses a critical epistemic gap in the scientific application of Explainable AI (XAI). In scientific domains (e.g., neuroscience, physics, biochemistry), opaque machine learning models are increasingly used not merely as predictors but as sources of insight into the underlying phenomena they model. The prevailing assumption is that if a model is sufficiently reliable (its predictions match the phenomenon's outcomes) and its post-hoc explanation is sufficiently faithful (the explanation accurately reflects the model's internal computations), then the explanation provides a justified claim about the structure of the phenomenon itself.

The authors argue that this inference is structurally invalid. While reliability and faithfulness are necessary conditions, they are insufficient to bridge the gap between "understanding the model" and "understanding the world." Specifically, neither metric verifies whether the model's internal structure corresponds to the causal, mechanistic, or counterfactual structure of the phenomenon.

Methodology and Formalism
The paper employs a formal epistemic analysis rather than empirical experimentation. It constructs a chain of approximation to analyze the inferential relationship between the phenomenon, the model, and the explanation:

The Phenomenon ( $f$ ): The true function representing the natural phenomenon.
The Model ( $h^*$ ): A hypothesis selected from a space $H$ (e.g., a deep neural network) that approximates $f$ .
The Explanation ( $g^*$ ): A representation generated by a post-hoc method $E$ (e.g., SHAP, LIME, attention maps) that approximates $h^*$ .

The chain is denoted as $f \xleftarrow{\phi} h^* \xleftarrow{E} g^*$ , where:

$\phi$ (the model–world link) represents the reliability of $h^*$ in tracking $f$ .
$E$ (the model-of-model link) represents the faithfulness of $g^*$ in describing $h^*$ .

The authors distinguish between two types of epistemic verdicts:

Descriptive Verdicts: Assessments of what the model computes (faithfulness).
Justificatory Verdicts: Assessments of the model's ability to track the phenomenon (reliability).

The core methodological argument is that a scientist seeks a third type of verdict: a justificatory claim about the structure of the phenomenon. The paper demonstrates that composing a descriptive verdict (about $g^*$ ) and a justificatory verdict (about $h^*$ ) cannot logically yield a justificatory verdict about $f$ 's structure.

Key Contributions and Arguments

The Structural Failure of Composition:
The paper argues that the gap between model and world does not close even in the limit of perfect reliability and perfect faithfulness.
- Reliability checks if outputs match outcomes but does not require shared internal structure. A model can perfectly track a phenomenon via spurious correlations or non-mechanistic shortcuts (e.g., relying on image borders rather than pathology).
- Faithfulness checks if the explanation matches the model's computation but is entirely internal to the model. It cannot determine if the model's computation corresponds to the phenomenon's mechanism.
- Consequently, a "perfect" chain ( $f \approx h^* \approx g^*$ ) still leaves the question of structural alignment unanswered. The chain lacks the premise of "shared structure" required to move from model-facts to world-facts.
Three Cases of Misleading Chains (Appendix A):
The authors illustrate the persistence of this gap through three scenarios:
- Case 1 (Distribution Shift): A model relies on a spurious feature (e.g., laterality markers in X-rays) that works in training but fails in new populations. Faithfulness correctly reports the reliance, but reliability fails only when the distribution shifts. The explanation remains faithful to the wrong mechanism.
- Case 2 (Invariant Confounds): A model relies on a spurious feature that is correlated with the target across all testable populations (e.g., patient positioning). Here, reliability is perfect everywhere, and faithfulness is perfect, yet the model still fails to capture the true mechanism. No amount of validation data can distinguish the spurious model from a mechanistic one if they produce identical predictions.
- Case 3 (Perfect Tracking, Divergent Structure): A model tracks the phenomenon perfectly on all inputs but uses an internal structure entirely different from the phenomenon's. A faithful explanation of this model describes a structure that is not the phenomenon's structure. Reliability and faithfulness are identical for both the "true structure" model and the "divergent structure" model, rendering the metrics incapable of distinguishing truth from luck.
Distinction Between "Strong" and "Weak" Readings:
- Strong Reading (How-Actually): Claims that the model's structure is the phenomenon's structure. The paper argues the chain cannot support this claim on its own.
- Weak Reading (How-Possibly/Objectual Understanding): Claims that the model offers a candidate hypothesis or a grasp of connections within the domain. The paper concedes that the chain can support this, provided the output is situated against external corroboration (prior theory, independent evidence, or experimental intervention).

Results and Claims
The paper does not present experimental results but offers a philosophical and logical proof of limitation:

No Self-Sufficient Justification: A reliable model paired with a faithful explanation does not, by composition alone, justify a claim about the phenomenon's causal or mechanistic structure.
External Corroboration is Required: To move from a description of the model to a claim about the world, external premises (independent theory, known mechanisms, or interventions) must be introduced. These premises are not generated by the XAI chain itself.
The Role of Idealization: The paper aligns with literature on scientific idealization (e.g., Sullivan, Elgin), suggesting that opaque models function as idealized tools. They can provide "objectual understanding" (grasping connections and possibilities) but not "explanatory understanding" (identifying the actual mechanism) without external validation.

Significance
The paper's significance lies in clarifying the epistemic boundaries of post-hoc XAI in science. It challenges the routine practice of treating faithful explanations of reliable models as direct windows into scientific reality.

It argues that the failure to distinguish between "how the model works" and "how the world works" is not a technical flaw to be solved by better algorithms or higher fidelity, but a structural limitation of the inferential chain.
It reorients the scientific use of XAI: rather than viewing explanations as definitive proofs of mechanism, they should be viewed as generators of candidate hypotheses that require rigorous external testing and theoretical integration.
The paper concludes that while the chain supports the generation of "how-possibly" claims, it cannot license "how-actually" claims without external corroboration, a distinction crucial for the integrity of scientific discovery using opaque models.