🤖 machine learning

The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations

This paper argues that the primary failure of existing time series attribution methods lies not in their scalar output format but in a fundamental computational mismatch where marginal conditioning and off-manifold gradients conflate direct and mediated temporal dependencies, rendering them unfaithful to the model's underlying causal structure.

Original authors: Amadeo Tunyi

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Amadeo Tunyi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Mystery of the Time-Traveling Detective

Imagine you are a detective trying to solve a crime, but the only clue you have is a single, blurry photo of the suspect. You know the suspect was at the scene, but you don't know how they got there, who they talked to along the way, or if they were even the one who actually pulled the trigger. In the world of Artificial Intelligence (AI), this is exactly the problem scientists face when they try to understand how computer models make predictions about the future. These models, which power everything from stock market forecasts to medical monitoring, are often "black boxes." We know what goes in (data from the past) and what comes out (a prediction), but we don't know the internal logic.

To fix this, researchers have developed "Explainable AI" (XAI). Think of XAI as a tool that tries to highlight the most important clues. For time-based data (like weather patterns or heart rates), the current standard method is like giving every single clue a score from 1 to 10. If a temperature reading at 2:00 PM gets a score of 9, the tool says, "This was super important!" But here's the catch: in the real world, events are connected. The temperature at 2:00 PM didn't happen in a vacuum; it was caused by the wind at 1:00 PM, which was caused by a storm at 12:00 PM. If your detective tool just gives a score to the 2:00 PM temperature without understanding that it was just a messenger for the storm, you might blame the wrong thing. This paper dives deep into why our current detective tools are failing to see the full picture of how time works.

The Paper: Why Our "Scorecards" Are Wrong

The paper, titled The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations, argues that the most popular way we try to explain AI predictions is fundamentally broken when it comes to time. The authors, led by Amadeo Tunyi, prove that the standard method—giving every data point a single "importance score"—is like trying to describe a complex movie by only listing the brightness of each frame. It misses the plot.

The Core Problem: The "Mediator" Trap
The authors use a clever analogy involving a chain of events. Imagine a domino setup:

You push the first domino (Event A).
It knocks over the second domino (Event B).
The second domino knocks over the third (Event C).

In this chain, Event A causes Event C, but only because it went through Event B. Event B is the "mediator." The paper shows that current AI explanation tools are terrible at telling the difference between a direct cause and a mediated one. They see that Event A happened and Event C happened, so they give Event A a high score, thinking it directly caused the result. But in reality, Event A only mattered because it moved Event B. The tools are "hallucinating" a direct connection that doesn't exist because they are ignoring the middleman.

The "Off-Manifold" Mistake
The paper also points out a second, sneaky error. Imagine you are walking on a tightrope (the "manifold"). This is the only path where the data makes sense. If you step off the rope into the void, you are in a place that never happens in real life. The authors found that many popular methods (like SHAP and Gradient-based tools) accidentally look at these "off-rope" scenarios. They ask, "What would happen if the temperature was 100 degrees when the humidity was 0%?" Even though that combination is impossible in the real world, the math calculates a score for it. This leads the AI to give importance to things that are mathematically possible but physically impossible, confusing the explanation.

What the Paper Proves
The authors didn't just guess; they ran simulations and mathematical proofs to show that:

Standard methods fail: Popular tools like SHAP, TimeSHAP, and Saliency maps consistently give credit to the wrong variables when data is "autocorrelated" (when today's data depends heavily on yesterday's).
They can't tell the difference: In their tests, these tools gave the same "importance score" to a variable whether it was a direct cause or just a messenger. They couldn't distinguish between a direct link and a chain reaction.
They break the rules of reality: By looking at impossible data combinations, they violate the "on-manifold" rule, meaning their explanations don't respect the actual laws of the data they are analyzing.

The Proposed Solution: The "Transition Tensor"
So, if the scorecard is broken, what do we use instead? The authors propose a new way of thinking called DAG-faithfulness. Instead of asking "How important is this single number?", we should ask "What is the map of connections?"

They suggest replacing the simple list of scores with a Transition Tensor. Imagine a 3D grid or a complex spreadsheet that doesn't just say "A is important." Instead, it maps out: "A at time 1 influences B at time 2, which then influences C at time 3."

It captures the direction of influence.
It captures the timing (the lag).
It captures the relationships between variables.

The paper argues that for an explanation to be truly faithful, it must be able to reconstruct this map of connections (a Directed Acyclic Graph, or DAG) exactly as the AI model learned it. If the explanation says "A connects to C," but the model actually learned "A connects to B, and B connects to C," the explanation is a lie, even if the final prediction was correct.

The Current State of Affairs
The authors are careful to note that while they have identified the problem and defined the perfect solution (DAG-faithfulness), they haven't fully built the perfect tool yet. They show that some newer methods (like FIT and WinIT) are closer to the mark because they try to respect the "rules of the road" (staying on the manifold), but they still struggle with sensitivity. The paper concludes that the field needs to stop trying to squeeze complex time-based logic into simple numbers and start building these 3D "connection maps."

In short, the paper is a wake-up call: we can no longer trust the simple "importance scores" for time-series AI. They are missing the story of how events connect, and until we switch to methods that map those connections, our explanations will remain incomplete and potentially misleading. The authors have drawn the map of the problem and pointed the way to the solution, but the journey to build the perfect map is just beginning.

Technical Summary: The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations

1. Problem Statement

The paper addresses a fundamental structural inadequacy in current Explainable AI (XAI) methods for time series models. While the dominant paradigm in XAI relies on flat attribution scoring—assigning a scalar value to each feature at each timestep to represent its "direct" contribution—this approach fails to capture the relational and conditional nature of temporal reasoning.

The authors argue that existing methods (including gradient-based, perturbation-based, and attention-based approaches) conflate direct temporal dependencies with mediated dependencies caused by autocorrelation. By collapsing high-dimensional relational structures into one-dimensional summaries, these methods discard the conditional dependency information that constitutes the model's actual reasoning process. Consequently, they cannot distinguish between a feature that directly influences a forecast and one that influences it only indirectly through an intermediate variable.

2. Methodology and Theoretical Framework

Definitions and Notation:

Flat Attribution (Definition 1): A method producing a scalar score $\phi_i^{(t)}(x)$ for a variable-time pair $(X_i, t)$ via marginal conditioning or gradients, without conditioning on the values of other inputs that may mediate the effect.
Model Dependency Graph (MDDAG, $G_f$ , Definition 2): A directed acyclic graph (DAG) representing the true dependencies learned by a model $f$ . An edge exists from input $X_i^{(t_1)}$ to forecast target $X_j^{(T+w)}$ if and only if the input is conditionally dependent on the forecast given all other inputs.
DAG-Faithfulness (Definition 3): A new faithfulness criterion requiring that the dependency graph $G_T$ induced by an explanation method encodes the same conditional independencies (Markov equivalence) as the MDDAG $G_f$ .

The Computational Gap:
The paper identifies a "computational gap" where flat methods fail to satisfy two necessary admissibility conditions:

On-manifold Evaluation (C1): The attribution must depend only on the model's behavior within the support of the data distribution ( $\text{supp}(p)$ ), not on off-manifold reparameterizations.
Conditional Separation (C2): The attribution to an input must vanish if that input is conditionally independent of the forecast given the remaining inputs.

Case Studies and Proofs:
The authors construct specific scenarios to demonstrate the failure of standard methods:

Scenario I (Autocorrelation): A model where $X^{(2)}$ mediates the effect of $X^{(1)}$ on the forecast. Flat methods (e.g., SHAP variants) incorrectly assign non-zero scores to $X^{(1)}$ because they fail to condition on $X^{(2)}$ , violating C2.
Scenario II (On-manifold constraints): A deterministic process where $X^{(2)} = \delta X^{(1)}$ . Here, $X^{(1)}$ has no direct effect on the forecast within the data support. However, gradient-based methods (Saliency, Integrated Gradients) detect non-zero derivatives because they probe off-manifold points, violating C1.

Proposed Representation:
To overcome these limitations, the paper proposes replacing scalar scores with a Transition Tensor (Definition 4). This is a three-dimensional array $T_{loc} \in \mathbb{R}^{D \times D \times T}$ (or $D \times D \times T \times W$ for multi-step forecasts) that quantifies the influence of source variable $i$ at lag $t_1$ on target variable $j$ . The global explanation is derived from the support of the aggregated global tensor, separating structural existence from strength magnitude.

3. Key Contributions

Formalization of the Computational Gap: The paper rigorously defines the mismatch between flat attribution methods and the conditional dependency structure of temporal models, proving that scalar scores cannot capture the necessary conditional independence structure.
Definition of DAG-Faithfulness: It introduces DAG-faithfulness as the appropriate criterion for global time series explanations, grounded in Markov equivalence to the model's learned dependency graph.
Proof of Failure for Standard Methods:
- Theorem 1: Proves that standard SHAP (and its time-series extensions like TimeSHAP) are not DAG-faithful in general, as they generate spurious edges due to autocorrelation.
- Proposition 1 & 2: Demonstrates that marginal influence-based methods (KernelSHAP, ShapTime) fail Conditional Separation (C2), while gradient-based methods (Saliency, Integrated Gradients) fail On-manifold Evaluation (C1).
Evaluation of Existing Extensions: The paper evaluates temporally-aware methods (FIT, WinIT, Dynamask). While they partially satisfy admissibility by using in-distribution replacements, they suffer from low sensitivity to genuine direct edges (false negatives), though they are structurally closer to admissibility than flat methods.
Proposal of Admissible Instantiations: The authors propose that a valid explanation must be a Transition Tensor instantiated via an admissible functional that satisfies both C1 and C2. They suggest combining manifold-aware Jacobians (for C1) with conditional distribution shift measures (for C2).

4. Results and Empirical Evidence

The paper provides empirical evidence through synthetic benchmarks comparing various methods against ground-truth dependency graphs:

Flat Methods (SHAP, TimeSHAP, ShapTime, TS-MULE): Consistently fail to distinguish between direct and mediated effects. In Scenario I, they assign non-zero scores to fully mediated variables. In Scenario II, they fail to respect on-manifold constraints.
Gradient Methods (Saliency, IG, TIG, SmoothGrad): Correctly distinguish direct vs. mediated effects in some cases but consistently fail Scenario II by assigning non-zero relevance to variables that are conditionally independent on the data manifold, due to off-manifold sensitivity.
Perturbation Methods (FIT, WinIT, Dynamask): These methods do not hallucinate edges in the failure scenarios (they respect C1 and C2 better) but exhibit low sensitivity, often assigning near-zero scores to genuine direct edges (false negatives).

No existing method tested achieves correctness across all scenarios, confirming the paper's claim that current paradigms are structurally inadequate.

5. Significance and Claims

The paper claims that the field of time series XAI must shift from output-centric faithfulness (does perturbing the feature change the output?) to structure-centric faithfulness (does the explanation recover the model's internal conditional independence structure?).

Theoretical Shift: The authors argue that the scalar format itself is not the primary failure mode, but rather the marginal conditioning and off-manifold evaluation inherent in current algorithms.
New Standard: They posit that DAG-faithfulness is the necessary criterion for global explanations, and that the Transition Tensor is the minimal representational object capable of encoding these dependencies.
Open Challenge: The paper identifies the construction of a concrete, admissible instantiation of the transition tensor that satisfies both C1 and C1 in practice as the central open problem for future research. It explicitly notes that while the tensor is the correct internal representation, visualization strategies are needed to make this high-dimensional structure human-interpretable.

The paper concludes by calling on the XAI community to adopt this structure-centric framework and move beyond flat attribution scores for temporal domains.