📊 statistics

Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression

This paper benchmarks tabular foundation models for conditional density estimation across 39 real-world datasets, demonstrating that they consistently achieve superior accuracy and log-likelihood compared to diverse baselines, particularly excelling in low-data regimes and even outperforming larger specialized models in specific applications like photometric redshift estimation.

Original authors: Rafael Izbicki, Pedro L. C. Rodrigues

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rafael Izbicki, Pedro L. C. Rodrigues

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster.

The Old Way (Point Prediction):
Most traditional machine learning models act like a basic weather app that just says, "Tomorrow will be 72°F." It gives you a single number. But what if there's a 50% chance of a blizzard and a 50% chance of a heatwave? The average is 72°F, but that number is useless to you. You need to know the whole picture: the risk of snow, the chance of rain, and how uncertain the forecast is.

The Goal (Conditional Density Estimation):
This paper is about teaching computers to stop just guessing a single number and instead draw a full "weather map" of possibilities. This is called Conditional Density Estimation (CDE). It answers: "Given these specific conditions (covariates), what is the entire shape of the possible outcomes?"

The New Contenders: "Tabular Foundation Models"

For a long time, to get this full "weather map," you needed to hire a specialist for every specific job.

Need to predict house prices? Use a "House Specialist."
Need to predict galaxy positions? Use a "Galaxy Specialist."
Need to predict stock volatility? Use a "Stock Specialist."

These specialists are great, but they require a lot of data to learn their specific trade.

Enter the Tabular Foundation Models (like TabPFN and TabICL). Think of these as super-interns who have read every book in the library before they even started their job.

They haven't been trained on your specific data yet.
Instead, they were pre-trained on millions of synthetic problems.
When you hand them your data, they don't need to "study" (re-train) from scratch. They just look at your data and say, "Ah, I've seen patterns like this before. Here is my best guess at the full distribution."

The Big Race (The Benchmark)

The authors of this paper organized a massive race to see who is better at drawing these "weather maps."

The Racers: The new "Super-Interns" (Foundation Models) vs. the old "Specialists" (traditional statistical and neural network methods).
The Track: 39 different real-world datasets (ranging from housing prices to galaxy positions).
The Conditions: They tested them with very little data (50 examples) and a lot of data (20,000 examples).

The Results: What Happened?

1. The "Super-Interns" are shockingly good at small data.
When the data was scarce (only 50 examples), the Foundation Models crushed the competition.

Analogy: Imagine a chess grandmaster (the Foundation Model) playing against a local club player (the Specialist). Even if the grandmaster hasn't seen this specific board setup before, their vast experience allows them to make a brilliant move immediately. The local player needs to study the board for hours to catch up.
Result: The Foundation Models produced the most accurate "weather maps" on the vast majority of datasets, even with tiny amounts of data.

2. The "Specialists" catch up when you give them a library.
When the data size grew to 20,000 examples, the Specialists started to catch up.

Analogy: If you give the local chess player a million hours to study the board, they can eventually beat the grandmaster.
Result: The gap narrowed. The Foundation Models were still usually the best, but the Specialists became very competitive.

3. The "Calibration" Glitch.
While the Foundation Models were great at drawing the shape of the map, they were sometimes a bit "overconfident" or "underconfident" about the odds.

Analogy: The Foundation Model might draw a perfect map of a storm, but it might say "There is a 90% chance of rain" when it's actually only 70%. The Specialists were sometimes better at getting the percentages right.
Fix: The paper suggests that if you take the Foundation Model's map and just tweak the percentages slightly (a process called "recalibration"), you get the best of both worlds.

The "Space Telescope" Test (SDSS Case Study)

To prove how powerful these models are, the authors ran a special test using data from the Sloan Digital Sky Survey (500,000 galaxies).

The Challenge: Predict the distance of galaxies based on their light.
The Setup: They let the "Specialists" train on the entire 500,000-galaxy dataset. They let the "Super-Intern" (TabPFN) train on only 50,000 galaxies (10% of the data).
The Outcome: The Super-Intern, using only 10% of the data, beat the Specialists who used 100% of the data.
Why it matters: In the real world, data is expensive and hard to get (like labeling galaxies). This proves that Foundation Models can save you massive amounts of money and time because they learn so much more from less data.

The Bottom Line

This paper tells us that we no longer need to build a custom, specialized machine learning model for every single problem.

We now have off-the-shelf "Universal Predictors" that can look at a table of data and instantly generate a high-quality, full probability map. They are:

Faster to start: No need for long training times.
Data efficient: They work wonders even with small datasets.
Accurate: They often beat the best specialized tools.

The only catch? They are like a super-fast sports car; they need a powerful engine (GPU) to run, and if the road (dataset) is too huge or too twisty (high dimensions), they might run out of gas (memory) before they finish. But for most everyday problems, they are the new champions.

1. Problem Statement

Conditional Density Estimation (CDE) aims to recover the full conditional distribution $f(y|x)$ of a response variable $y$ given tabular covariates $x$ , rather than just predicting the conditional mean or quantiles. This is critical in scenarios involving:

Heteroscedasticity: Variance changes with inputs.
Multimodality: Multiple distinct outcomes for a single input.
Asymmetric Uncertainty: Skewed predictive distributions.

While recent Tabular Foundation Models (e.g., TabPFN, TabICL) have demonstrated state-of-the-art performance in point prediction (regression/classification) via in-context learning, their effectiveness as general-purpose CDE methods has not been systematically evaluated. Existing benchmarks focus heavily on point metrics (RMSE, R²), leaving a gap in understanding whether these models can accurately capture complex distributional shapes and maintain calibration compared to purpose-built CDE methods.

2. Methodology

2.1 Models Evaluated

The study benchmarks three variants of tabular foundation models against a diverse set of baselines across five methodological families:

Foundation Models (No fine-tuning, pure in-context learning):

TabPFN-2.5: Outputs a discrete bar distribution (histogram) over the response range.
RealTabPFN-2.5: A variant fine-tuned on real-world tabular data, also outputting bar distributions.
TabICL-Quantiles: Outputs predictive quantiles directly.
- Post-processing: Bar distributions are converted to densities via linear interpolation and renormalization. Quantiles are interpolated to form a CDF, then numerically differentiated to obtain densities.

Baselines (Purpose-built CDE methods):

Parametric: Gaussian (homoscedastic/heteroscedastic), Student-t, Log-normal, and Gamma GLMs (with and without ridge regularization).
Quantile-based: Quantile-Tree (Gradient Boosted Trees at 49 quantile levels).
Regression-based: FlexCode-RF (Cosine basis expansion coefficients estimated via Random Forests).
Tree-based Probabilistic: BART (Homo and Hetero variants using XBART).
Neural Density Estimators: Mixture Density Networks (MDN), Flow-Spline (Conditional Normalizing Flows), and CatMLP (Discretized MLP with Softmax).

2.2 Experimental Setup

Datasets: 39 real-world regression datasets from OpenML and SDSS DR18 (Astrophysics), covering domains like housing, robotics, bioinformatics, and finance.
Sample Sizes: Evaluated across nested subsamples of $n \in \{50, 500, 1k, 5k, 10k, 20k\}$ . A specific scaling study was conducted on the SDSS dataset up to $n=500,000$ .
Dimensions: Covariate dimensions ( $d$ ) ranged from 5 to 563.
Evaluation Metrics: Six complementary metrics were used:
1. CDE Loss: A proper scoring rule minimizing the expected squared difference between the estimated and true density.
2. Log-Likelihood: Cross-entropy of the predictive distribution.
3. CRPS (Continuous Ranked Probability Score): Measures the difference between the predictive CDF and the true outcome.
4. PIT KS Statistic: Kolmogorov–Smirnov test on Probability Integral Transform values to assess calibration.
5. 90% Coverage: Empirical coverage of predictive intervals.
6. Computation Time: Wall-clock time for fitting and prediction.
Protocol: 5 independent train/test splits (50 splits for $n=50$ ). Baselines underwent hyperparameter tuning via cross-validation; foundation models used default pre-trained settings with no fine-tuning.

3. Key Contributions

First Broad Benchmark for Tabular CDE: The paper provides the first systematic evaluation of tabular foundation models specifically for conditional density estimation, moving beyond point prediction metrics.
Large-Scale Empirical Design: Covers 39 datasets, sample sizes up to 20,000 (and 500,000 in a case study), and dimensions up to 563, using a rigorous multi-metric evaluation framework.
Diagnostic Analysis of Failure Modes: Identifies specific scenarios where foundation models underperform (e.g., discrete/quasi-discrete responses) and analyzes the trade-off between sample efficiency and calibration.
Scaling Case Study: Demonstrates that foundation models trained on a fraction of the data can outperform classical methods trained on the full dataset.

4. Key Results

4.1 Density Accuracy (CDE Loss, Log-Likelihood, CRPS)

Dominance: Foundation models consistently achieved the best CDE loss, log-likelihood, and CRPS across the vast majority of datasets.
- At $n=50$ : Foundation models won on 82% of datasets.
- At $n=1,000$ : Foundation models won on 92% of datasets.
- At $n=20,000$ : Foundation models won on 75% of datasets.
Sample Efficiency: Foundation models showed superior performance even at very small sample sizes ( $n=50$ ), often capturing complex structures (like bimodality) where non-foundation methods required significantly more data.
TabICL Nuance: While TabICL-Quantiles performed exceptionally well on CRPS (which can be computed directly from quantiles), it lagged slightly on CDE loss and log-likelihood. This is attributed to the noise introduced during the numerical differentiation step required to convert quantiles to densities.

4.2 Calibration

Small Samples: Foundation models were competitive or superior in calibration at small sample sizes ( $n \le 1,000$ ).
Large Samples: As sample size increased ( $n \ge 5,000$ ), foundation models' calibration ranks deteriorated (mid-table), while task-specific neural baselines (like MDN) often achieved better calibration. This suggests that in-context learning is highly effective for density shape but less optimal for probabilistic calibration at scale compared to end-to-end training.

4.3 Computational Efficiency

Speed: Foundation models were significantly faster than complex neural baselines (e.g., Flow-Spline, MDN) because they require no training (only inference).
Memory Constraints: TabPFN variants encountered Out-of-Memory (OOM) errors on high-dimensional datasets ( $d=384$ ) at $n=20,000$ , whereas TabICL-Quantiles handled these cases successfully.

4.4 Case Study: Photometric Redshift (SDSS)

Scenario: Estimating redshift distributions for 500,000 galaxies.
Finding: TabPFN trained on only 50,000 galaxies (10% of the data) outperformed all baselines trained on the full 500,000-galaxy dataset across CDE loss, CRPS, and log-likelihood.
Implication: Foundation models offer a massive data-efficiency advantage, potentially reducing data requirements by an order of magnitude for uncertainty-aware regression tasks.

4.5 Failure Modes

Foundation models were occasionally outperformed by specialized baselines in specific regimes:

Discrete/Quasi-Discrete Responses: On datasets like "Digits" or "BlackFriday," models like CatMLP (which explicitly discretize the output) or parametric models with strong inductive biases performed better, especially at small sample sizes.
High Dimensions + Large N: Memory constraints limited the applicability of TabPFN variants in high-dimensional settings without subsampling.

5. Significance and Conclusion

This paper establishes tabular foundation models as strong "off-the-shelf" conditional density estimators.

Practical Impact: They offer a plug-and-play solution that often surpasses complex, task-specific CDE pipelines without requiring hyperparameter tuning or dataset-specific training.
Data Efficiency: Their ability to extract predictive signal from small datasets makes them ideal for domains where labeled data is expensive or scarce (e.g., astrophysics, medical imaging).
Future Directions: The authors suggest that combining foundation models with post-hoc recalibration techniques could address their calibration gaps at large sample sizes. Furthermore, extending these evaluations to multivariate and structured outputs is a natural next step.

In summary, while not a panacea for every distributional shape (particularly discrete supports), tabular foundation models represent a significant leap forward in making high-quality conditional density estimation accessible and efficient for tabular data.