🤖 machine learning

High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

This paper reveals that while Tabular Foundation Models outperform Gradient-Boosted Decision Trees in predictive accuracy, they suffer from inferior uncertainty quantification and calibration, highlighting a critical performance-reliability trade-off that must be addressed for their trustworthy adoption.

Original authors: José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of detectives to solve a mystery based on a list of clues (the "tabular data"). Some detectives are old-school veterans who have solved thousands of cases using a specific, reliable method (like Gradient Boosted Decision Trees, or GBDTs). Others are brand-new, super-smart AI detectives trained on millions of fake cases before they ever saw a real one (these are the Tabular Foundation Models, or TFMs).

This paper is a report card comparing how well these two types of detectives perform, but with a twist: it doesn't just ask, "Who got the most answers right?" It also asks, "Who knows when they are guessing?"

Here is the breakdown of their findings:

1. The "Smart but Overconfident" Trap

The new AI detectives (TFMs) are incredibly fast and accurate. When asked to pick the right suspect, they get the highest score (AUC). They seem like the future of detective work.

However, the researchers found a hidden flaw. When these AI detectives are unsure, they don't admit it. Instead, they confidently point to a single suspect, even when the clues are messy or confusing. They are like a student who guesses the answer on a test with 100% confidence, even though they actually have no idea what they are talking about.

2. The "Safe but Honest" Veteran

The old-school detectives (GBDTs) are slightly less accurate overall. They might miss a few more clues than the AI. But, they are much better at knowing their limits. When the clues are confusing, they say, "I'm not sure, it could be any of these five people."

In the world of this paper, this honesty is measured by a concept called Conformal Prediction. Think of this as a "safety net."

The Goal: If you want to be 90% sure you caught the right person, your "safety net" (the list of suspects you provide) should contain the real culprit 90% of the time.
The Result: The old-school detectives built nets that actually worked 90% of the time. The new AI detectives built nets that looked small and precise, but they missed the real culprit more often than they should have. They were "overconfident."

3. The Trade-Off: Speed vs. Safety

The paper calls this a "Performance–Uncertainty Trade-off."

The AI (TFMs): High performance, low reliability. They are great when the data is clean and simple, but they get shaky and overconfident when the data is noisy or messy.
The Veterans (GBDTs): Slightly lower performance, but high reliability. They stay calm and honest even when the clues are confusing.

4. The "Synthetic" Stress Test

To be sure, the researchers created a fake, chaotic crime scene with lots of noise and confusing clues.

The AI detectives got the highest score but made huge, confident mistakes.
The veteran detectives were more consistent. They admitted uncertainty when the clues didn't make sense, making them safer to trust in tricky situations.

The Bottom Line

The paper concludes that while these new AI models are amazing at getting the "right answer" on clean data, they haven't yet learned how to be humble. They are currently high-performance but low-reliability.

If you need a detective for a simple, clear-cut case, the AI is a great choice. But if you are dealing with a messy, complex, or high-stakes situation where being wrong is dangerous, the old-school, honest detectives (GBDTs) are still the safer bet because they know when they are guessing.

In short: The new AI is the fastest runner, but it trips over its own shoelaces when the track gets bumpy. The old runner is a bit slower, but they never stumble when the path gets rough.

Technical Summary: High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

Problem Statement
Recent Tabular Foundation Models (TFMs), such as TabPFN and TabICL, have demonstrated state-of-the-art predictive performance on small and medium-scale tabular datasets, often surpassing traditional Gradient-Boosted Decision Trees (GBDTs). However, the trustworthiness of these models, specifically regarding their uncertainty quantification, remains largely unexplored. While predictive accuracy (AUC) is the primary metric for current benchmarks, the reliability of model confidence is critical for safety-critical applications in finance and healthcare. Existing literature focuses almost exclusively on accuracy, leaving a gap in understanding whether TFMs provide well-calibrated uncertainty estimates or if they exhibit overconfidence, particularly under noisy or complex conditions.

Methodology
The authors propose a comprehensive evaluation protocol that extends beyond standard accuracy metrics to include uncertainty quantification via Conformal Prediction (CP). The study evaluates four families of models across 112 small and medium tabular datasets from the TALENT benchmark, supplemented by 20 synthetic datasets designed to isolate high-noise and low-separability regimes.

Models Evaluated:
- TFMs: TabPFN, TabICL, PFN-v2, and Mitra.
- GBDTs: XGBoost, LightGBM, and CatBoost.
- Baselines: Classical methods (k-NN, Logistic Regression) and Deep Learning (MLP, TabM).
Conformal Prediction Framework:
- The study utilizes the Least Ambiguous Class (LAC) score, $s(x, y) = 1 - \hat{p}(y|x)$ , to transform point predictions into prediction sets.
- A calibration set (disjoint from training data) is used to compute nonconformity scores and determine a quantile threshold $q_\alpha$ for a target coverage of $1-\alpha = 0.90$ .
- This ensures distribution-free guarantees that the true label lies within the prediction set with probability $\geq 1-\alpha$ .
Evaluation Metrics:
- AUC: Standard predictive performance.
- Coverage Rate (CR): The proportion of test samples where the true label is in the prediction set.
- Average Set Size (SS): Measures the informativeness of the prediction sets (smaller sets imply higher confidence).
- Size-Stratified Coverage Score (SSCS): A critical metric introduced to assess conditional coverage. It measures empirical coverage across groups of samples with similar set sizes. A low SSCS indicates that the model is overconfident (producing small sets) when it is actually wrong, a failure of calibration that marginal coverage might miss.

Key Results
The empirical results, averaged over 15 seeds, reveal a distinct performance–uncertainty trade-off:

Predictive Performance vs. Reliability: TFMs achieve the highest AUC scores (e.g., TabICL at 0.890, PFN-v2 at 0.889) compared to GBDTs (e.g., LightGBM at 0.864). However, TFMs exhibit significantly lower SSCS scores (ranging from 0.494 to 0.517) compared to GBDTs (ranging from 0.532 to 0.544).
The Trade-off: There is an inverted trend between accuracy and uncertainty quality. While TFMs are more accurate, their confidence does not faithfully reflect predictive uncertainty. They tend to produce "overconfident errors"—narrow prediction sets that fail to include the true label.
Synthetic Data Findings: On synthetic datasets with high noise and weak class separability, the gap widens. TFMs produce high-variance, overconfident predictions, whereas tree-based models (GBDTs) remain robust and maintain higher conditional coverage (SSCS of 0.840 for GBDTs vs. 0.614 for TFMs in synthetic tests).
Baseline Behavior: Classical Logistic Regression showed good marginal calibration but struggled with conditional coverage near non-linear boundaries. Deep learning baselines (MLP, TabM) generally performed worse than TFMs in both accuracy and calibration.

Significance and Claims
The paper concludes that while TFMs advance the frontier of predictive accuracy, they currently suffer from a major reliability deficit regarding uncertainty quantification. The authors claim that:

TFMs are not yet ready for high-stakes deployment where calibrated uncertainty is required, as they systematically yield lower size-stratified coverage scores than traditional models.
GBDTs remain the most reliable choice for small-to-medium datasets, particularly those with high noise, heterogeneity, or complex feature distributions, offering better calibration and robustness despite slightly lower AUC.
Future Work: The authors identify that achieving well-calibrated uncertainty is a major open challenge. They suggest future research should explore distribution shifts, model-specific uncertainty mechanisms (e.g., ensembling, Bayesian heads), and alternative pretraining distributions to reduce overconfidence.

The study provides the first systematic assessment of TFM uncertainty, demonstrating that high performance does not inherently guarantee high reliability in tabular learning.