💻 computer science

Estimand-Relative Validity and Context-Driven Instability of Shapley Explanations for Tabular Foundation Models

This paper demonstrates that the validity and stability of Shapley explanations for tabular foundation models are critically dependent on the choice of removal operator relative to the intended estimand and are significantly influenced by in-context examples, necessitating explicit specification of these factors for reliable interpretation.

Original author: Emin Talip Demirkiran

Published 2026-11-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original author: Emin Talip Demirkiran

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a growing class of models designed to learn from examples rather than by memorizing fixed rules. Imagine a student who, instead of studying a textbook for years, is handed a single page of notes and a new question, then asked to solve it immediately using only that page. This is how modern "foundation models" for tabular data work. They take a small set of labeled examples, called a context, and a new query, and they make a prediction without ever changing their internal code. Because these systems are so flexible, researchers have long tried to understand why they make the choices they do. They use a method called Shapley values, which acts like a fair division of credit. If a prediction is a team effort, this method calculates how much each piece of information contributed to the final result. However, to figure out how much a specific piece of information matters, the method must first imagine what would happen if that piece were missing. This creates a puzzle: when you remove a piece of data, how do you fill the empty space? Do you replace it with an average, with a blank, or with a new example that fits the pattern? The answer to this question changes the entire story of how the model thinks.

A recent study by Emin Talip Demirkiran at Eskisehir Technical University tackles this puzzle head-on, focusing on two of the most advanced models of this type, known as TabPFN and TabICL. The researcher wanted to know if the choice of how to fill in missing data changes the explanation of the model's decision, and whether those explanations remain stable when the surrounding details shift slightly. To find out, the team ran a massive, carefully planned series of tests using both synthetic data, where the correct answers were known by design, and real-world datasets. They tested five different ways of handling missing information, ranging from simple averaging to more complex methods that ask the model to re-evaluate the situation with the missing piece removed. The results revealed a striking truth: there is no single "best" way to explain these models. The method that produces the most accurate explanation for one type of question is often the worst for another.

The study found that the choice of method depends entirely on what question you are trying to answer. If you want to know how the model would behave if you intervened and changed a specific value, the most accurate explanations came from methods that simply replaced the missing data with averages or standard placeholders. However, if you want to know how the model reasons based on the specific relationships within the data it was shown, the best explanations came from methods that asked the model to re-contextualize itself with the missing piece removed. In fact, the researchers observed that the method that performed best for one goal often performed the worst for the other. This means that an explanation is not a fixed property of the model; it is relative to the specific question being asked. Without specifying exactly how the missing data was handled and what kind of answer is desired, an explanation cannot be trusted.

Beyond the question of which method to use, the study also investigated how stable these explanations are when the environment changes. In these models, the prediction depends heavily on the specific examples chosen to form the context. The researchers tested whether changing the order of the columns, the way labels were coded, or the random seed used to start the model would alter the explanation. They found that these factors had almost no effect. Instead, the explanation was overwhelmingly dominated by which specific examples were drawn into the context. In other words, the explanation is highly sensitive to the specific set of examples the model is looking at, but remarkably stable against other technical variations. This suggests that the reliability of an explanation is tied to the specific context provided, not just the model itself.

The researchers also tested a safety check by adding a feature to the data that contained no useful information at all, essentially a "dummy" variable. They expected the model to ignore this feature and assign it zero importance. Instead, both models frequently changed their predictions in response to this useless feature, assigning it significant weight. This indicates that these powerful models can be easily swayed by noise, and their explanations can reflect this sensitivity. The study concludes that for these types of AI systems, an explanation is not a standalone fact. It is a fragile construct that depends on the specific method used to handle missing information, the specific question being asked, and the specific examples the model is currently considering. To understand these models, we must stop looking for a single, universal explanation and start asking exactly what kind of explanation we need and under what conditions it holds true.

Technical Summary: Estimand-Relative Validity and Context-Driven Instability of Shapley Explanations for Tabular Foundation Models

Problem Statement
Tabular foundation models (TFMs), such as TabPFN and TabICL, utilize in-context learning to generate predictions without parameter updates. While Shapley values are a standard method for post-hoc feature attribution in these models, their validity is contingent upon the choice of a "removal operator"—the mechanism used to evaluate predictions when features are absent. In conventional models, this choice is already a recognized source of disagreement; however, for TFMs, the implications are more complex. Because TFMs consume an entire context at inference, removing a feature may involve altering the query, the context, or both. Furthermore, the specific "estimand" (the target quantity being estimated, e.g., interventional vs. conditional) dictates which removal operator is theoretically appropriate. The author notes that the effects of different operators on attribution validity and stability in in-context models remain insufficiently understood.

Methodology
The study employs a preregistered, factorial experimental design to evaluate five distinct removal operators across two TFM architectures (TabPFN v2 and TabICL v2). The methodology is structured around two primary hypotheses:

H1 (Operator Correctness): The study compares operators against ground-truth reference attributions derived from synthetic data generated via Gaussian copulas with known target functions. The operators evaluated are:
- O1: Marginal imputation (averaging over context rows).
- O2: Baseline imputation (replacing with context means).
- O3: Missing-value encoding (replacing with NaN).
- O4: Recontextualisation (restricting the context and query to the feature subset).
- O5: Conditional estimation via in-context learning (fitting a regressor on a disjoint pool).
The author computes exact Shapley values (rather than sampling approximations) for 12 dependence regimes and compares the resulting attributions against two reference games: an interventional game ( $v_{int}$ ) and a conditional game ( $v_{cond}$ ).
H2 (Stability): The study investigates attribution instability under "nuisance" variations common in in-context learning. A fully crossed design (48 configurations) tests the variance contribution of four factors:
- Context draw (sampled in-context examples).
- Column order.
- Label coding.
- Model seed.
Attribution vectors are standardized, and a crossed random-effects ANOVA is used to decompose variance shares ( $\pi_F$ ) for each factor.

Key Contributions

Ground-Truth Comparison: A preregistered comparison of five removal operators across two TFMs using exact Shapley enumeration and synthetic data with known target functions, avoiding reliance on other explainers for validation.
Estimand-Relative Validity: Evidence demonstrating that operator correctness is strictly dependent on the chosen estimand. The operator that best approximates the interventional game performs poorly for the conditional game, and vice versa.
Variance Decomposition: A quantitative analysis showing that attribution instability is overwhelmingly dominated by the sampled in-context examples, while column order, label coding, and model seeds contribute negligible variance.
Negative Control Results: A study-wide finding that both models frequently alter predictions in response to "planted-null" features (features constructed to contain no label information), challenging standard invariance assumptions.

Results

Operator Performance: The choice of removal operator materially changes attribution correctness.
- For the interventional estimand, marginal imputation (O1) and mean imputation (O2) yielded the lowest errors.
- For the conditional estimand, recontextualisation (O4) and conditional estimation via ICL (O5) yielded the lowest errors.
- The ranking of operators is reversed between the two estimands. For example, in TabICL v2 under the conditional game, O4 and O5 are superior, whereas O1 and O2 are inferior.
- Feature dependence ( $\rho$ ) further modulates these rankings; as dependence increases, the performance gap between operators shifts.
Stability: The variance decomposition reveals that the context draw accounts for the vast majority of attribution variance (approx. 91–98% across models). In contrast, column order, label coding, and model seed contribute negligible variance (all < 0.3%).
Null Feature Sensitivity: Both models demonstrated sensitivity to features containing no label information, often changing their predictions in response to these null features, indicating a lack of invariance in standard controls.

Significance and Claims
The paper concludes that Shapley explanations for tabular foundation models cannot be considered interpretable or valid without explicitly specifying both the removal operator and the intended estimand. The study demonstrates that there is no single "best" operator; rather, the optimal choice is contingent on whether the goal is to approximate an interventional or conditional quantity.

Furthermore, the author asserts that context sensitivity must be reported as a fundamental component of explanation reliability. The finding that the specific draw of in-context examples dominates attribution variance suggests that explanations generated by TFMs are inherently unstable relative to the context, a factor that practitioners must account for. The study limits its claims to the two specific binary-classification foundation models tested and low-dimensional settings via exact enumeration, cautioning against extrapolating these specific rankings to larger feature spaces or other model families without further validation.

Technical Summary: Estimand-Relative Validity and Context-Driven Instability of Shapley Explanations for Tabular Foundation Models

More like this