Technical Summary: MulTaBench

Problem Statement

Tabular Foundation Models (TFMs) have recently established state-of-the-art performance in supervised tabular learning by leveraging pretraining on structured numerical and categorical data. However, these models are fundamentally unimodal; they lack native support for unstructured modalities like text and images. Current approaches to Multimodal Tabular Learning (MMTL) typically rely on frozen, pre-trained embeddings (e.g., from LLMs or VLMs) to process unstructured inputs before feeding them into tabular learners.

The paper argues that this approach is suboptimal for two primary reasons:

1. Information Loss: Pre-trained embeddings are optimized for broad, global semantic content (e.g., distinguishing an X-ray from a mammogram) and act as lossy summaries. They often discard fine-grained details (e.g., precise size estimations, localized anomalies, or specific textual nuances) that are critical for specific downstream predictive tasks.
2. Task-Agnostic Limitations: Existing benchmarks often focus merely on the co-occurrence of modalities rather than the necessity of Target-Aware Representations (TAR). In many MMTL tasks, the optimal representation of an unstructured modality depends heavily on the specific prediction target and the context provided by other modalities. Current benchmarks fail to isolate datasets where task-agnostic embeddings fail to capture the necessary predictive signal.

Methodology

The Curation Pipeline

To address the gap in benchmarking, the authors developed an algorithmic pipeline to curate datasets that satisfy two specific properties:

1. Joint Signal: The unstructured modality (text or image) must provide complementary information that improves predictive performance when combined with structured tabular data, beyond what either modality achieves alone.
2. Task-Awareness: The dataset must demonstrate that Target-Aware Representations (TAR)—embeddings tuned specifically to the prediction target—outperform frozen, off-the-shelf embeddings.

Experimental Conditions:
The pipeline evaluates candidate datasets using five diverse tabular learners (LightGBM, CatBoost, TabM, TabPFNv2, TabPFN-2.5) under four conditions:

• Unimodal Structured: Tabular features only.
• Unimodal Unstructured: Unstructured features only (via embeddings).
• Joint Frozen: Tabular + Unstructured (using frozen embeddings).
• Joint TAR: Tabular + Unstructured (using embeddings fine-tuned on the target).

TAR Implementation:
To generate TAR, the authors fine-tune the last three layers of the encoder (using LoRA) on the prediction target as a preprocessing step. This adaptation is performed independently of the structured features and the downstream tabular learner. For regression tasks, continuous labels are discretized into bins to stabilize training.

Acceptance Criteria:
A dataset is included in the benchmark only if, for at least 3 out of 5 learners:

• The Joint Frozen condition outperforms both unimodal baselines (verifying Joint Signal).
• The Joint TAR condition outperforms the Joint Frozen condition (verifying Task-Awareness).

MulTaBench Dataset

Using this pipeline, the authors constructed MulTaBench, a benchmark comprising 40 datasets:

• 20 Image-Tabular datasets: Curated from public sources and literature, spanning domains like healthcare (e.g., CheXpert, CBIS-DDSM), e-commerce, and biology. This represents the largest image-tabular benchmark to date.
• 20 Text-Tabular datasets: Derived from existing benchmarks (e.g., CARTE, TextTabBench, AutoML Multimodal), filtered to meet the strict curation criteria.
• Diversity: The benchmark balances classification and regression tasks, varying sample sizes (400 to 114,000 rows), and feature counts.

Key Results

1. Superiority of Target-Aware Representations

The core finding is that TAR consistently outperforms frozen embeddings across the MulTaBench datasets.

• Generalization: The performance gains from tuning embeddings generalize across different tabular learners (from GBDTs to TFMs), encoder scales (Small vs. Large models), and embedding dimensions (15, 30, 60, or raw 384 dimensions).
• Magnitude of Gain: While the magnitude of improvement varies by dataset, the trend is robust. For instance, in the text-tabular subset, TAR improved performance over frozen embeddings for all tested learners.
• Qualitative Evidence: Attention map analysis (using DINO-v3) on image datasets (e.g., CheXpert, PetFinder) reveals that TAR shifts the encoder's focus from irrelevant global features to specific, task-relevant regions (e.g., lung tissue for pneumonia detection, animal ears for age prediction), whereas frozen embeddings remain scattered or focused on task-irrelevant details.

2. Limitations of Existing Benchmarks

When the authors applied their curation pipeline to existing text-tabular benchmarks (56 unique datasets from sources like CARTE and TextTabBench), they found that:

• Approximately 23% failed the Joint Signal criterion (the unstructured modality added no value).
• Of the remaining, 36% failed the Task-Awareness criterion (frozen embeddings were sufficient).
• Only 41% of the original pool passed both criteria.
  This suggests that many existing benchmarks do not effectively isolate the specific challenges that require novel multimodal architectures, potentially masking the benefits of TAR.

3. Robustness and Computational Costs

• Encoder Scale: Increasing the embedding model size (from Small to Large) improved downstream performance, but TAR still significantly outperformed frozen embeddings even at the larger scale. Notably, a tuned Small model often outperformed a frozen Large model.
• Dimensionality: The gains from TAR were not artifacts of PCA compression; they persisted even when using raw 384-dimensional embeddings.
• Cost: TAR introduces significant computational overhead, primarily due to the fine-tuning step. Text TAR is notably more expensive than Image TAR due to the larger effective dataset size (multiple text columns).

Significance and Claims

The paper positions MulTaBench as a necessary tool for the development of the next generation of Multimodal Tabular Foundation Models.

• Bridging the Gap: The authors argue that current architectures (including PFNs and joint modeling approaches like TabSTAR or AutoGluon-Multimodal) have not successfully maintained SOTA tabular performance while learning TAR for text and images. MulTaBench explicitly targets datasets where this capability is required.
• Future Directions: The benchmark is designed to facilitate research into architectures that combine the contextualization benefits of TAR with the robustness and efficiency of In-Context Learning (ICL). The authors suggest that future models should ideally leverage the best of both worlds, potentially by coupling TFMs with LLMs/VLMs or by expanding synthetic priors to include multimodal features.
• Modest Claims: The authors acknowledge that their curation pipeline entangles the computational problem with the algorithmic solution (selection bias), making it difficult to predict a priori if a new dataset will pass. However, they assert that the pipeline successfully isolates a distinct class of MMTL tasks where static representations fail, providing a rigorous testbed for evaluating true multimodal foundation models.

In summary, MulTaBench demonstrates that for high-impact multimodal tabular tasks, task-specific adaptation of unstructured representations is not just beneficial but often necessary, and it provides the first large-scale, rigorously curated benchmark to drive research in this direction.