Technical Summary: Towards Evaluating Data Priors for Tabular Foundation Models

Problem Statement

Tabular foundation models (TFMs), such as TabPFN, rely heavily on data-generating priors to define the task distribution during pretraining. While these priors are central to shaping downstream predictive behavior, they are rarely evaluated as independent components. Typically, a prior is introduced alongside specific architectural choices, optimization strategies, and training protocols, making it difficult to isolate the specific impact of the prior itself. This lack of isolation hinders the ability to compare priors from different models or understand how data-level properties translate to downstream performance. The paper addresses the methodological question: How can priors from different tabular foundation models be compared independently of the architectures and training protocols they were originally introduced with?

Methodology

To address this, the authors propose a unified evaluation pipeline that standardizes data generation, model training, and downstream evaluation. This framework is built as an extension of the TFM-Playground.

1. Unified Prior Interface

The authors implemented a unified interface to generate training tasks from five distinct prior families:

• TabPFNv1: Uses function-based priors (MLPs, Gaussian processes, and mixtures) to sample target functions on random inputs.
• TabICL: Generates tasks by sampling latent Gaussian variables and propagating them through layered Structural Causal Model (SCM) transformations. Variants include MLP mappings, tree-based mappings, or mixtures.
• TICL: Similar to TabPFNv1 but uses alternative preprocessing and target conversion routines based on MLP and Gaussian process generators.
• TabForestPFN: Fits standard ML models (decision trees, nearest neighbors, random projection classifiers) to randomly sampled data and uses their predictions on unseen inputs as targets.
• Real Data: A controlled pipeline using curated OpenML datasets. It involves dataset curation, leakage reduction (removing duplicates via statistical fingerprinting), and task construction where datasets are partitioned into classification/regression pools. Tasks are generated by sampling datasets and selecting targets (either original or random compatible columns).

2. Controlled Experimental Setup

• Task Generation: Each prior generates a fixed budget of 250,000 tasks. Tasks are constrained to a maximum of 512 samples and 16 features to ensure computational feasibility.
• Model Architecture: All priors are used to train an identical nanoTabPFN model. This lightweight architecture is chosen to retain competitive performance in low-data regimes while allowing for controlled, repeatable experiments under fixed resources.
• Training Protocol: A fixed protocol is applied across all priors, including a batch size of 1 to handle varying sequence lengths without padding.
• Downstream Evaluation: Models are evaluated on a fixed subset of TabArena v0.1 classification tasks (16 tasks with ≤500 features and ≤5000 instances). Performance is measured via ROC-AUC across three folds.

3. Evaluation Metrics

The study compares priors using two dimensions:

1. Data-Level Characterization: Generated tasks are summarized via a shared vector of dataset statistics (meta-features) to compute pairwise similarity in data space.
2. Downstream Performance: Priors are ranked based on absolute performance (Raw Avg. ROC-AUC) and relative consistency (Norm. Avg. min-max normalized ROC-AUC and average rank).

Key Results

Performance Variability

Different priors favor different downstream behaviors, with no single prior dominating all metrics:

• Absolute Performance: The TICL prior achieved the highest raw average ROC-AUC (0.866) and the most dataset wins (5).
• Relative Consistency: The TabICL Mix SCM prior achieved the best average rank (2.69) and the highest normalized average (0.937), indicating more consistent competitiveness across diverse tasks compared to others.
• Real Data Performance: Surprisingly, Real Random Targets (randomly selecting target columns from real datasets) outperformed Real Default Targets (using original targets) significantly (0.675 vs. 0.123 normalized average). The authors attribute this to task diversity; random selection allows a single dataset to generate multiple distinct prediction problems, whereas fixed targets may lead to repetitive, similar problems.

Data Similarity vs. Performance

• Data Space: Priors from the same library (e.g., TabICL variants) are generally more similar to each other than to priors from other libraries, suggesting the generation framework influences the data distribution more than the specific underlying mechanism.
• Performance Space: While some data-space similarities correlate with performance similarities (e.g., TabPFN variants), the correlation is not perfect.
  • Dissimilar Data, Similar Performance: The TabForest Neighbor prior is statistically very dissimilar to real data yet achieves competitive downstream performance.
  • Similar Data, Different Performance: Real Random Targets and Real Default Targets are statistically similar in data space, yet their downstream performance differs drastically.
• Conclusion: Data-level statistical similarity is an informative but incomplete proxy for downstream predictive behavior. Factors such as task diversity and learning signals are not fully captured by summary statistics.

Ablation Study

An ablation study on data dimensionality revealed that increasing the number of features while keeping the sample size fixed generally led to worse downstream performance. The authors suggest this creates an unrealistic feature-to-sample ratio compared to the evaluation tasks.

Significance and Claims

The paper claims to take a "first step" towards exploring how data-generating priors affect downstream behavior under a controlled setting. Its primary contributions are:

1. Isolation of Priors: Successfully isolating priors as independent components by fixing the architecture and training protocol, a task rarely attempted in previous TFM literature.
2. Unified Interface: Providing a standardized interface for comparing synthetic and real-data priors across multiple existing frameworks (TabPFN, TabICL, TabForest, etc.).
3. Empirical Insights: Demonstrating that while data-level similarity offers some insight, it does not fully explain downstream behavior. The study highlights that task diversity (e.g., via random target selection) may be a critical factor in the success of real-data priors, independent of statistical similarity to the source data.

The authors remain modest, noting that the observed performance differences are often small in magnitude and should be interpreted cautiously. They suggest that future work could examine whether prior-specific hyperparameter tuning might alter these relative rankings.