Technical Summary: CoMET - Modular Multimodal Classification Without Fine-Tuning

Problem Statement

Multimodal classification tasks typically require complex, end-to-end training pipelines that involve fine-tuning large foundation models on specific datasets. This approach is computationally expensive, time-consuming, and often necessitates dataset-specific hyperparameter tuning. While Tabular Foundation Models (TFMs) have recently demonstrated the ability to perform strong predictive tasks via in-context learning without fine-tuning, their application has been largely restricted to tabular data or required fine-tuning when applied to multimodal settings. The core challenge addressed by this paper is how to leverage the "out-of-the-box" nature of TFMs for multimodal classification (combining text, images, and tabular data) without any task-specific training or backpropagation, while overcoming the mismatch between standard modality encoders (which often output fixed-size tokens like CLS) and the TFM's input requirements.

Methodology: CoMET

The authors propose CoMET (Composing Modality Encoders with Tabular foundation models), a modular pipeline that composes frozen pre-trained backbones with a TFM. The method operates in four distinct steps:

1. Feature Extraction: Each non-tabular modality (e.g., image, text) is passed through a frozen, pre-trained backbone (e.g., DINOv3 for images, ELECTRA for text) to extract sample-wise token-level embeddings.
2. Pooling: Token embeddings are aggregated into fixed-size vector representations. The paper introduces PALPooling (Pseudo Attention Label Pooling) as a lightweight, adaptive alternative to standard CLS tokens or mean-pooling.
   • PALPooling Mechanism: Instead of backpropagation, PALPooling fits a linear scoring vector via ridge regression. It uses "Pseudo Attention Labels" (PALs) derived from the TFM's own predicted label distributions. Specifically, it splits the training data into a support set and a query set. The TFM predicts on the query set, and a scoring function (based on Jensen-Shannon Divergence between predictions and prior label frequencies) generates targets for a ridge regression model. This model learns to weight tokens to maximize the shift from the prior distribution, effectively identifying informative tokens without gradient descent through the TFM.
3. Dimensionality Reduction (PCA): The pooled embeddings are compressed using Principal Component Analysis (PCA). The authors demonstrate that PCA is not merely a compression tool but a crucial adaptor that aligns the geometry of the frozen encoder's embeddings with the TFM's pre-training prior (which is synthetic and high-rank).
4. Prediction: The compressed, concatenated multimodal features (tabular + projected non-tabular) are fed into the TFM (specifically TabICLv2) along with a support set of labeled examples. The TFM performs in-context learning to predict the label distribution for the query sample.

Key Contributions

1. PCA as a Sufficient Adaptor: The paper demonstrates that applying PCA to embeddings from frozen backbones is sufficient to make TFMs competitive multimodal classifiers. This projection significantly improves performance by increasing the effective rank of the embeddings, aligning them with the TFM's synthetic pre-training distribution, and balancing signals between tabular and non-tabular features.
2. CoMET Framework: An easy-to-implement, modular framework that achieves state-of-the-art (SOTA) results on diverse multimodal benchmarks without any fine-tuning. Components (encoders, TFMs) can be swapped independently.
3. PALPooling: A novel, training-free adaptive pooling method. It avoids the need for backpropagation through the TFM by using pseudo-labels derived from the model's own outputs. It consistently outperforms CLS tokens and mean-pooling, particularly in scenarios where CLS tokens fail to capture task-relevant features (e.g., images with multiple objects).
4. Scalable Hierarchical Classification: The authors showcase the application of CoMET to hierarchical classification (H-CoMET). By decomposing large classification problems (e.g., >500k samples, >2,000 classes) into smaller sub-tasks within a tree structure, the method enables fast, scalable inference without training separate models for each subtask.

Results

The paper evaluates CoMET across unimodal and multimodal benchmarks:

• Unimodal Performance: On image and text datasets, CoMET with PCA and PALPooling matches or exceeds the performance of linear probing and multi-layer perceptrons (MLPs), often outperforming them significantly (e.g., +3.4% to +6.2% on ImageNet subsets).
• Multimodal Benchmarks: CoMET achieves competitive or SOTA results on datasets combining text, images, and tabular data (e.g., Jigsaw Toxicity, PetFinder, MM-IMDb, Airbnb). It outperforms previous multimodal TFM approaches like MMPFN, which required fine-tuning with modality-specific projectors.
• Hierarchical Scaling: On large-scale hierarchical datasets like iNaturalist (2,896 classes) and Amazon MM, H-CoMET achieves high accuracy (82.50% on iNaturalist) with inference times under 6 minutes, significantly outperforming flat classifiers and matching or beating SOTA hierarchical text models.
• Efficiency: The method requires no training time for the TFM or encoders. PALPooling fitting takes seconds (e.g., 16s to 54s for image datasets).

Significance and Claims

The paper argues that the composition of foundation models offers a simple, powerful, and "out-of-the-box" solution for multimodal learning. Its primary significance lies in challenging the necessity of complex, end-to-end training pipelines for new problems.

• Modularity: The approach allows for the independent replacement of backbones or TFMs as they improve, inheriting future progress without retraining.
• Training-Free Efficiency: It demonstrates that strong predictive performance can be achieved by amortizing Bayesian inference via in-context learning, even for non-tabular data, provided the embeddings are properly adapted (via PCA) and pooled (via PALPooling).
• Scalability: The method enables the handling of massive datasets and fine-grained class spaces (thousands of classes) that are often intractable for traditional fine-tuning approaches due to computational constraints.

The authors maintain a modest stance, acknowledging that while PCA is a simple adaptor, more sophisticated TFM-informed dimensionality reduction methods could yield further gains. They also note that the current work focuses on text and image modalities but the template is generalizable to other domains with off-the-shelf encoders (e.g., audio, video).