🤖 machine learning

Do Tabular Foundation Models Agree with Themselves?

This paper reveals that Tabular Foundation Models (TFMs) fail to produce faithful joint distributions because they violate both marginalization and factorization consistency requirements across all evaluated datasets and tasks.

Original authors: Christian Klötergens, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme, Tom Hanika

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Christian Klötergens, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme, Tom Hanika

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for clues in a crime scene, you are looking for clues in a giant spreadsheet. This is the world of Tabular Foundation Models, a hot new area of artificial intelligence where computers learn to predict things based on rows and columns of data, like predicting if a loan will be paid back or what the price of a house might be. These models are built on a clever idea called Bayesian inference, which is basically a fancy way of saying, "Let's update our beliefs based on new evidence." Think of it like a weather forecaster who starts with a general guess about rain, but as soon as they see dark clouds, they update their prediction to say, "Okay, it's definitely going to rain."

For a long time, scientists hoped these AI models were perfect "weather forecasters" for data. They believed that if you asked the model to predict one thing, then use that prediction to guess another, the whole story it told would fit together perfectly, just like a well-written novel where every chapter connects logically to the next. But here's the catch: in the real world, we often don't know the "true" answer to check against. It's like trying to grade a student's essay without having the answer key. So, instead of asking, "Is this model right?", a group of researchers asked a simpler, trickier question: "Does this model even tell a consistent story to itself?" They wanted to know if the AI was just making up random numbers that happened to look good, or if it was actually following the strict rules of logic and probability.

The paper you're about to hear about, titled "Do Tabular Foundation Models Agree with Themselves?", dives right into this mystery. The researchers, led by Christian Klötergens and his team, decided to put the most popular AI models for tabular data (like TabPFN, TabICL, and TabFM) through a rigorous "logic test." They didn't just ask the models to predict; they asked them to tell the same story in two different ways and checked if the stories matched.

Imagine you are trying to guess the flavor of a mystery smoothie.

Test 1 (Marginalization Consistency): You ask the AI, "What flavor is this smoothie?" Then, you ask it, "If the smoothie is red, what flavor is it? And if it's blue, what flavor is it?" Finally, you mix those two answers together based on how likely the smoothie is to be red or blue. If the AI is consistent, this "mixed" answer should be exactly the same as the first direct answer.
Test 2 (Factorization Consistency): You ask the AI to tell the story of the smoothie's ingredients in two different orders. First, it guesses the fruit, then the sweetener. Second, it guesses the sweetener, then the fruit. If the AI is truly understanding the data, the final picture of the smoothie should be identical, no matter which ingredient it guessed first.

The researchers ran these tests on a huge variety of real-world datasets, from wine quality ratings to car prices and medical records. They used a mathematical ruler called Total Variation Distance to measure how far apart the answers were. A score of 0 means the model is perfectly consistent; a higher score means the model is contradicting itself.

The results were a bit of a shocker. Every single model they tested failed both tests. On every dataset, for both simple yes/no questions (classification) and number-predicting questions (regression), the models gave different answers depending on how you asked the question. It's as if the AI told you the smoothie was "strawberry" when asked directly, but when you asked about the color first, it suddenly decided it was "blueberry," even though the color was the same.

The paper found that while some models were slightly better at telling a consistent story than others (for example, a model called TabFM was the most consistent classifier, and TabPFNv3 was the regressor that came closest to consistency), none of them were truly consistent. In fact, the inconsistencies were so widespread that the researchers concluded these models do not actually represent a single, unified "joint distribution" of the data. In plain English: the models are not following the strict laws of probability that they are supposed to be mimicking. They are essentially "hallucinating" different versions of reality depending on the order in which you ask them questions.

The authors also discovered something interesting: being a "good" predictor (getting the right answer most of the time) doesn't mean you are a "consistent" predictor. A model could be very accurate at guessing the price of a house but still contradict itself when you ask it to explain its reasoning in a different order. This suggests that consistency is a separate, hidden flaw that current benchmarks miss.

So, what does this mean for the future? The paper doesn't say these models are useless—they are still the best tools we have for many tasks. However, it sounds a loud alarm that we cannot blindly trust these models to tell us about complex relationships between variables if we need them to be logically coherent. The researchers suggest that future models need to be built with "consistency" as a core feature, perhaps by training them to penalize these contradictions, rather than just focusing on getting the final answer right. Until then, we have to remember that these powerful AI "detectives" are great at spotting clues, but they sometimes tell different versions of the story depending on who's asking.

Title: Do Tabular Foundation Models Agree with Themselves?

Problem Statement
Tabular Foundation Models (TFMs), such as TabPFN, TabICL, and TabFM, currently represent the state-of-the-art for tabular prediction. These models are constructed as transformers trained to approximate the Bayesian posterior predictive distribution (PPD) under a synthetic prior. While their predictive heads are univariate, practitioners utilize them to model multivariate distributions by autoregressively sampling targets and appending them as features. However, the faithfulness of the resulting joint distribution has not been investigated. Crucially, because the ground-truth distribution of real-world datasets is unknown, TFMs cannot be evaluated against the true Bayesian posterior. This leaves a gap in understanding whether the univariate predictions of these models actually cohere into a single, well-defined joint distribution.

Methodology
To address the inability to measure distance from a true posterior, the authors propose a label-free approach: determining whether a model's predictions could have been induced by any joint distribution at all. A well-defined joint distribution imposes strict mathematical constraints on its marginals and conditionals. The authors formalize two consistency requirements that any model induced by a valid joint distribution must satisfy:

Marginalization Consistency: This requires that marginalized conditionals equal directly predicted marginals. Specifically, if a model predicts a target $A$ directly, and also predicts $A$ conditioned on a second target $B$ (which is then marginalized out using the model's own prediction for $B$ ), the resulting distributions must be identical. This is an instance of the law of total probability and the martingale property of coherent predictive systems.
Factorization Consistency: This is a stronger requirement concerning the joint distribution itself. It demands that different autoregressive factorization orders (e.g., predicting $A$ then $B$ , versus $B$ then $A$ ) result in the exact same joint distribution.

The authors prove that factorization consistency implies marginalization consistency, but the converse is not true. To evaluate these properties, they derive metrics based on the Total Variation (TV) distance between the distributions generated by the model under these different conditions. These metrics function without knowledge of the ground-truth distribution or the pretraining prior.

Experimental Setup
The study evaluates several state-of-the-art TFMs (TabPFNv2/v3, TabICLv1/v2, TabDPTv1.2, and TabFM) across a broad suite of datasets from OpenML, covering both classification and regression tasks. For each dataset, two target columns are selected that plausibly depend on each other given the remaining features. The models are tested under 5-fold cross-testing to measure the discrepancy between:

Direct marginal predictions vs. marginalized mixtures (Marginalization Consistency).
Joint distributions derived from different sampling orders (Factorization Consistency).

Key Results
The experiments reveal systematic and significant violations of both consistency requirements across all evaluated models, datasets, and task types:

Universal Violation: Every TFM evaluated violates both marginalization and factorization consistency on every dataset. The observed inconsistency is a structural property of current TFMs.
Magnitude of Violation: The violations are not merely sampling noise but represent structural gaps where the model's estimate of a target changes depending on whether an intermediate target is conditioned upon or marginalized.
Model Comparisons:
- Classification: TabFM (the newest model) demonstrates the highest consistency among classifiers, achieving the lowest average TV distance on the majority of datasets. TabICLv1 is identified as the most inconsistent classifier.
- Regression: No single model emerges as the most consistent across all datasets. TabDPT appears relatively consistent for marginalization but is among the most inconsistent for factorization. TabPFNv3 is the most consistent regressor regarding factorization.
Correlation: There is a strong positive correlation between the violations of marginalization and factorization consistency. However, there is no strong correlation between a model's predictive accuracy (or RMSE) and its consistency; high-performing models are not necessarily more coherent.
Feature Count: Contrary to intuition, wider tables (more features) do not necessarily lead to greater inconsistency; in classification tasks, consistency gaps were often higher on datasets with fewer features.

Significance and Contributions
The paper makes three primary contributions:

Formulation of Requirements: It defines two necessary conditions (marginalization and factorization consistency) for a TFM's implied joint distribution to be well-defined.
Label-Free Metrics: It derives metrics to quantify the extent of violations without requiring ground-truth labels or knowledge of the pretraining prior.
Empirical Evaluation: It applies these metrics to show that current state-of-the-art TFMs fail to produce coherent joint distributions.

The authors conclude that the univariate predictions of current TFMs do not arise from any single joint distribution over the table columns. Consequently, these models cannot be trusted for downstream tasks that rely on the dependence structure between targets, such as autoregressive data generation. The paper positions consistency as a distinct evaluation axis, separate from accuracy, that is currently missing from benchmarks like TabArena. While the conditions are currently diagnostic, the authors suggest that future work should focus on model design—such as training objectives that penalize consistency gaps or architectures that enforce a shared latent joint—to build TFMs that are provably coherent.

More like this