🤖 machine learning

The Importance of Encoder Choice:A Tabular-Image Study

This study evaluates state-of-the-art tabular models as encoders in image-tabular multimodal learning for the first time, addressing the challenge of applying in-context learning methods without access to test labels to demonstrate the critical importance of encoder selection.

Original authors: Ilia Koloiarov, Diego Coello de Portugal Mecke, Vijaya Krishna Yalavarthi, Tom Hanika, Lars Schmidt-Thieme

Published 2026-07-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ilia Koloiarov, Diego Coello de Portugal Mecke, Vijaya Krishna Yalavarthi, Tom Hanika, Lars Schmidt-Thieme

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate "detective bot" to solve mysteries. This bot needs to look at two types of clues: a photo of the scene and a spreadsheet of facts (like temperature, time, or names). To make this bot smart, you need a special "translator" for each clue type to turn them into a language the bot understands.

For the photos, we have amazing translators (like the famous ViT models). But for the spreadsheets? For years, scientists have been using a very simple, basic translator called an MLP (a plain, shallow neural network). It's like using a bicycle to deliver a pizza when you have a fleet of sports cars available. The paper argues that this is a huge mistake because the bicycle might be the reason the pizza arrives cold, not the delivery route itself.

The Big Discovery: The "Translator" Matters More Than You Think

The authors of this study decided to test this theory. They swapped out the boring bicycle (the plain MLP) for some of the most advanced, state-of-the-art "sports cars" available for spreadsheets. These new translators are called In-Context Learning Tabular Foundation Models (a mouthful, so let's call them ICL Super-Translators).

Here is the twist they found: The ranking of which detective bot is "best" changes completely depending on which translator you use.

If you test your bot with a weak translator, a fancy, complex detective method might look like a genius. But if you switch to a strong translator, that same fancy method might look average, while a simple, boring method suddenly looks like a superhero. The paper suggests that for years, we might have been praising the wrong methods simply because we were using bad translators.

The "Ghost in the Machine" Problem

There was a major hurdle with these new ICL Super-Translators. These models work by looking at a "context" (a list of solved examples with answers) to guess the answer for a new "query" (a mystery without an answer).

The authors discovered a spooky glitch: The model sees the same piece of information differently depending on whether it's in the "context" or the "query."

Imagine you are describing your favorite movie to a friend.

Context Mode: You are describing it after you've already seen the ending. You know the plot twists.
Query Mode: You are describing it before you've seen the ending. You are guessing what happens next.

Even though it's the same movie, your description (the "embedding") is totally different because your knowledge state is different. The paper found that when these models put training data in the "context" role and test data in the "query" role, they end up in two different neighborhoods in the computer's memory. It's like trying to match a map of New York with a map of London because the model got confused about which city it was looking at.

What the paper rules out:
The authors explicitly tested and ruled out the idea that this difference was just because the data was different. They showed that even when the exact same data point was used in both roles, the model still treated them differently. This is a structural flaw in how these models work, not a data problem.

The "Vanilla" Trap

Because of this glitch, the authors tested three ways to feed data to these models:

Vanilla (The "Naive" way): Just feed the training data as context and the test data as queries.
LOFO (Leave-One-Fold-Out): A complex way of shuffling data so everything gets a turn being a query.
NP (Non-Partitioned): A clever trick where the model sees the training data as both context and query at the same time.

The Verdict: The "Vanilla" way is a disaster. The paper shows that using the naive method consistently makes the bot perform worse. It's like trying to drive a sports car with the parking brake on. The authors strongly suggest avoiding this method entirely. Instead, the NP (Non-partitioned) method is the recommended "key" to unlock the full power of these models.

Do We Need Fancy Fusion?

For a long time, researchers thought they needed incredibly complex, expensive systems to combine photos and spreadsheets. These systems often have millions of parameters and require special pre-training.

The paper found something surprising: On datasets where both photos and spreadsheets are useful, a simple "bilinear fusion" (a basic math trick) paired with a strong translator performs just as well as the super-complex systems.

In fact, the fancy complex systems used, on average, 13.2 times more parameters than the simple baseline. The authors suggest that if you use a strong enough translator, you don't need the expensive, over-engineered fusion methods. The simple approach is just as good and much cheaper.

The "One-Modality" Warning

The paper also warns about datasets where one clue type is useless.

If you have a dataset where the spreadsheet is just noise (like the CCD dataset), adding the spreadsheet to the photo actually hurts the bot's performance.
If the photo is useless (like in Petfinder, where the spreadsheet is king), adding the photo hurts.

The authors found that the fusion module doesn't magically learn to "ignore" the useless clue. Instead, it gets confused and performs worse than if you had just used the good clue alone. This suggests that blindly combining data isn't always the answer; sometimes, you just need to pick the right clue.

How Sure Are We?

The authors didn't just guess; they ran extensive experiments across seven different real-world datasets (ranging from medical skin images to art auctions and car listings).

They measured the performance using F1 scores (a standard way to measure accuracy).
They used statistical tools (like OLS regression and Spearman correlation) to prove that the "lift" from combining data shrinks as the translator gets better.
They confirmed the "context-query shift" using MMD (Maximum Mean Discrepancy), a mathematical way to measure how far apart two groups of data are.

The paper concludes that the choice of translator is a "critical confound." This means that if you don't pick the right translator, your entire experiment might be misleading. The findings are based on measured data, not just simulations, and the authors are confident that the "Vanilla" extraction method should be avoided and that the "NP" method is the way to go for most cases.

So, the next time you build a detective bot, remember: Don't just focus on the detective; make sure your translator isn't wearing a blindfold.

Technical Summary: The Importance of Encoder Choice: A Tabular-Image Study

Problem Statement

Multimodal learning typically requires a dedicated encoder for each modality. While image and text encoders have evolved significantly, the tabular modality in multimodal settings has largely relied on plain, shallow Multi-Layer Perceptrons (MLPs). The authors argue that this reliance persists because the tabular domain has historically lacked strong deep learning candidates; Gradient-Boosted Decision Trees (GBDTs) have dominated tabular benchmarks but do not produce continuous representations compatible with multimodal pipelines.

Recent advances in In-Context Learning Tabular Foundation Models (ICL TFMs) offer strong, plug-in tabular encoders without task-specific retraining. However, a critical obstacle arises when using these models: the context-query representation shift. In ICL TFMs, training instances are processed as "context" (with labels visible), while test instances are processed as "queries" (without labels). Prior work (Ye et al., 2025) observed that this causes embeddings of the same instance to occupy different regions of the latent space depending on their role, violating the independent and identically distributed (i.i.d.) assumption required for downstream models. It was previously unknown whether this shift affects other ICL models or how it impacts multimodal fusion performance.

Methodology

The study conducts a systematic evaluation of tabular encoders in a tabular-image classification setting.

Experimental Setup:

Encoders: The authors evaluate three classes of tabular encoders:
- ICL TFMs: TabPFNv2, TabDPT, and TabICLv2.
- Non-ICL Foundation Models: TARTE.
- Baselines: TabM (pretrained from scratch), TabM-SSL (self-supervised), and TabVec (non-learned preprocessing).
Feature Extraction Schemes: For ICL models, three extraction methods are tested to address the context-query shift:
1. Vanilla ( $\star$ ): Training instances encoded as context.
2. Leave-one-fold-out ( $\dagger$ ): Training instances encoded as queries against the complement fold.
3. Non-partitioned ( $\wr$ ): All training instances passed simultaneously as both context and queries, with representations extracted in the query role.
Multimodal Fusion: A simple bilinear fusion module is used as the primary multimodal method, compared against a carefully engineered state-of-the-art method (TIP) and unimodal baselines.
Datasets: Seven real-world tabular-image datasets spanning medical imaging, art, vehicles, and pets (e.g., DVM, HAM10000, WikiArt, Petfinder).
Metrics:
- Relative Percentile Rank (RPR): Measures performance relative to an image-only baseline.
- Excess Distribution Shift (EDS): Quantifies the distributional gap between training and test embeddings using Maximum Mean Discrepancy (MMD), subtracting intrinsic data variation.

Key Contributions and Findings

1. Encoder Choice Dictates Multimodal Rankings

Multimodal performance rankings are not stable across different tabular encoders. A carefully engineered multimodal method (TIP) can outperform a simple baseline with a weak encoder but be matched or surpassed by the same simple baseline when paired with a strong encoder. Conclusions drawn from a single encoder do not generalize.

2. Fusion Can Degrade Performance in Modality-Dominated Datasets

In datasets where one modality carries the majority of predictive signal (e.g., image-dominated or tabular-dominated), combining modalities can degrade performance below the unimodal baseline. This suggests that fusion modules do not automatically learn to suppress uninformative modalities.

3. Encoder Quality Inflates Observed Fusion Advantages

In likely multimodal datasets, the observed performance lift from fusion is inversely related to encoder quality. As the tabular encoder improves, the marginal gain from fusion shrinks. This implies that weak encoders may artificially inflate the perceived benefit of multimodal fusion by compensating for poor unimodal representations.

4. Simple Baselines Match Complex Methods with Strong Encoders

On likely multimodal datasets, a simple bilinear fusion baseline paired with a strong encoder (e.g., TabPFN with Non-partitioned extraction) performs on par with TIP, a complex method requiring 13.2 times more parameters and a dedicated pretraining stage.

5. The Context-Query Shift is Universal to ICL TFMs

The representation shift is not unique to TabPFNv2; it is observed in TabDPT and TabICLv2 as well.

Vanilla ( $\star$ ) extraction consistently degrades performance across all three ICL models and should be avoided.
Non-partitioned ( $\wr$ ) extraction is the recommended scheme for most cases, effectively mitigating the shift and outperforming Leave-one-fold-out ( $\dagger$ ).

Significance

The paper argues that the choice of the tabular encoder is a critical, often overlooked confounding factor in multimodal learning. The study highlights that:

Benchmarking Stability: Conclusions about the efficacy of multimodal fusion are unstable if the tabular encoder is not carefully selected or if the encoder's representation shift is ignored.
Practical Efficiency: Strong, pre-trained tabular encoders can enable simple fusion architectures to match the performance of complex, engineered pipelines, reducing the need for massive parameter counts and pretraining stages.
Methodological Correction: The study provides empirical evidence that the "Vanilla" extraction method for ICL models is detrimental and establishes Non-partitioned extraction as a robust alternative for multimodal integration.

The authors conclude that future multimodal research must account for encoder quality and representation shifts to accurately assess the true value of combining modalities.