🤖 machine learning

LUCoS: Latent Unsupervised Context Selection for Tabular Foundation Models

The paper proposes LUCoS, an unsupervised method for selecting labeled instances in low-label tabular learning by leveraging the latent geometry of unsupervised embeddings rather than unreliable raw feature spaces, which achieves state-of-the-art performance across 67 datasets by ensuring effective coverage and representation quality.

Original authors: Oroel Ipas, Guillermo Gomez-Trenado, Rocío Romero-Zaliz, Isaac Triguero

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oroel Ipas, Guillermo Gomez-Trenado, Rocío Romero-Zaliz, Isaac Triguero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blank Canvas" Dilemma

Imagine you are a chef (the AI model) who is incredibly talented at cooking, but you only have a tiny amount of ingredients (labeled data) to work with. You have a massive pantry full of raw, unmarked vegetables and spices (the unlabeled data).

In the world of Tabular Foundation Models (like TabPFN), the chef doesn't learn by cooking a thousand meals and adjusting the recipe. Instead, the chef learns by In-Context Learning (ICL). This means the chef looks at a small "tasting menu" of a few examples (the context set) and immediately figures out how to cook the rest of the meal based on those examples.

The Catch: The chef is extremely sensitive to which examples you put on that tasting menu.

If you give the chef 5 random examples, they might guess the recipe is "Spicy."
If you give the chef 5 carefully chosen examples, they might guess "Sweet and Savory."
Both sets have the same number of items, but one leads to a delicious meal, and the other leads to a disaster.

The paper asks: How do we pick the best 5 examples to show the chef when we don't know the answers (labels) yet? This is called the "cold-start" problem.

The Old Way: Guessing in the Dark

Usually, when people try to pick these examples without knowing the answers, they look at the raw data.

The Analogy: Imagine trying to pick the best 5 fruits from a crate by looking at them in a dark room where the lights are off and the fruits are mixed up (apples next to rocks, bananas next to soup cans).
The Problem: Tabular data is messy. It has numbers, categories, missing values, and weird scales. Measuring "distance" between two rows of data in this raw state is like trying to measure the distance between a rock and a banana using a ruler meant for soup. It doesn't make sense.
The Result: The paper found that if you just pick random fruits or try to pick "different" fruits based on this messy view, you often do worse than just picking them completely at random.

The Solution: LUCoS (The "Magic Translator")

The authors propose a new method called LUCoS (Latent Unsupervised Context Selection).

Step 1: The Magic Translator (The Embedding)
Instead of looking at the raw, messy data, LUCoS uses a special "translator" (an unsupervised AI model called TabClustPFN).

The Analogy: This translator takes all the messy fruits and vegetables and rearranges them into a perfectly organized, high-tech warehouse. In this new warehouse, similar items are grouped together naturally. Apples are near apples, bananas near bananas, and rocks are far away from soup.
Why it works: This "latent space" (the new warehouse) creates a geometry where "distance" actually makes sense. Items that are close together in this new space are actually similar in a way that matters for the task.

Step 2: The Smart Picker (K-Medoids)
Once the data is in this organized warehouse, LUCoS uses a simple geometric rule to pick the examples.

The Analogy: Imagine you need to pick 5 representatives to show the chef. In the organized warehouse, you simply pick the 5 fruits that are most "central" to their groups. You pick the apple that is right in the middle of the apple pile, the banana in the middle of the banana pile, etc.
The Rule: This is called K-Medoids. It ensures you have good coverage of the whole warehouse without picking duplicates.

Step 3: The Labeling
You take those 5 specific fruits from the warehouse, map them back to the real world, and ask an expert to label them (e.g., "This is an apple"). Now you have your "tasting menu."

Step 4: The Prediction
You feed this perfect tasting menu to the TabPFN chef. The chef looks at these high-quality examples and predicts the rest of the data with amazing accuracy.

What the Experiments Showed

The authors tested this on 67 different datasets (like healthcare records, financial data, etc.) with very few labels (from 1 example per category up to 32).

The "Oracle" Test: They first proved that if you could magically know the answers and pick the absolute best 5 examples, the model would perform much better. This proved there was a huge "gap" to fill.
LUCoS vs. Random: LUCoS closed a significant chunk of that gap without ever seeing the labels.
The "Rescue" Effect:
- At very low budgets (1-2 examples): The simple act of picking any structured examples (coverage) helped.
- At medium budgets (4-16 examples): This is where LUCoS shined. The old method (picking from the messy raw data) actually started to fail and performed worse than random guessing. LUCoS, however, kept performing well because it was using the "organized warehouse" (the latent space).
- The Takeaway: The paper found that the geometry of the space matters more than the complexity of the picker. Using a simple picker in a smart space (LUCoS) beats a complex picker in a dumb space.

Summary in One Sentence

LUCoS solves the problem of picking the best training examples for AI by first translating messy data into a clean, organized "mental map" where similar things naturally group together, allowing a simple geometric rule to pick the perfect examples for the AI to learn from, even when no labels are available yet.

Technical Summary: LUCoS (Latent Unsupervised Context Selection)

Problem Statement

In low-label tabular learning, selecting which instances to label is a critical challenge. Recent Tabular Foundation Models (TFMs), such as TabPFN, rely on In-Context Learning (ICL), where predictive performance is highly sensitive to the specific labeled instances chosen as the "context set." While supervised oracle experiments demonstrate that carefully selected context sets can significantly outperform random selection under the same labeling budget, the cold-start setting—where instances must be selected before any labels are available—remains largely unaddressed in TFM literature.

Current unsupervised instance selection methods typically operate in the original tabular feature space. However, tabular data lacks a natural metric due to heterogeneous feature types, mixed scales, sparsity, and nonlinear interactions. Consequently, geometric distances in the raw space are often unreliable for predictive similarity, causing selection methods to perform inconsistently or even worse than random sampling as the labeling budget increases.

Methodology: LUCoS

The authors propose LUCoS (Latent Unsupervised Context Selection), a four-stage pipeline designed to perform instance selection without access to labels. The core hypothesis is that replacing the unreliable geometry of the original feature space with a meaningful latent geometry induced by an unsupervised TFM enables simple geometric selection methods to identify informative context sets.

The framework consists of two main components:

Representation Map ( $\phi$ ): Unlabeled training instances are embedded into a high-dimensional latent space using the encoder of an unsupervised Prior-Fitted Network (PFN). Specifically, the authors utilize the PIN Encoder from TabClustPFN, which generates 512-dimensional embeddings independent of cluster assignments. This step creates a task-relevant representation space where Euclidean distances better reflect predictive similarity.
Geometric Selector ( $\sigma$ ): Representative instances are selected in this latent space using K-Medoids clustering with Euclidean distance. The number of medoids ( $K$ $K$ ) is fixed to the labeling budget. K-Medoids is chosen because:
- It selects real training instances (medoids) rather than synthetic centroids, satisfying the requirement that selected samples be annotatable.
- It minimizes an explicit coverage objective, ensuring the selected subset covers the data distribution.
- It is deterministic (up to initialization) and requires no class priors.

Pipeline Flow:

Embed unlabeled training data into the latent space via the unsupervised PFN encoder.
Apply K-Medoids to select $K$ representative medoids in the latent space.
Map selected indices back to the original feature space and query an oracle (expert) for labels.
Use the resulting labeled subset as the in-context set for a supervised TFM (TabPFN-2.5) to predict on unseen test data.

Key Contributions

Empirical Validation of Sensitivity: The authors demonstrate that TFM performance in low-label regimes is highly sensitive to the specific context instances selected. A supervised oracle analysis (using a Steady-State Memetic Algorithm) reveals a substantial performance gap (up to $\approx 0.19$ $\Delta$ AUC at the smallest budgets) between random selection and optimized selection, indicating significant headroom for improvement.
LUCoS Framework: The proposal of a principled, label-free pipeline that decouples representation learning from selection. By performing selection in the latent space of an unsupervised PFN, LUCoS enables simple geometric methods (K-Medoids) to identify high-performing subsets.
Comprehensive Benchmarking: Evaluation on 67 OpenML-CC18 datasets across six low-label budgets ( $1C$ to $32C$ , where $C$ is the number of classes). LUCoS achieves the first rank in mean AUC, Accuracy, and F1 across all budgets.
Mechanism Decomposition: The paper decomposes LUCoS's gains into selector and representation components:
- At the smallest budgets ( $1C$ ), the gain is driven primarily by the selector effect (enforcing coverage via K-Medoids), with the latent space contributing little beyond raw features.
- From mid-range budgets ( $4C$ ) onward, the representation effect becomes dominant. In this regime, selection in the original feature space degrades (often falling below random), while the latent space "rescues" performance. This "rescue" effect is observed in the majority of datasets (78% of cases).

Results

Performance: LUCoS consistently outperforms baselines, including Random sampling, original-space K-Medoids, RDSS, and ZCore. It ranks first across all six labeling budgets.
Comparison to Oracle: LUCoS recovers approximately 13–20% of the gap between random selection and the supervised oracle, despite operating without any label information.
Robustness: The results are stable under leave-one-dataset-out analysis, progressive dataset removal, and across different evaluation metrics (AUC, ACC, F1).
Failure Modes of Alternatives: Original-space K-Medoids and other unsupervised methods (RDSS, ZCore) fail to reliably beat random selection, particularly as the budget grows, confirming that the geometry of the representation space is the critical bottleneck.

Significance and Claims

The paper argues that reliable unsupervised context selection for tabular data depends less on the sophistication of the selection algorithm and more on defining representativeness within a meaningful representation geometry.

The authors claim that:

The "headroom" for improvement in few-shot tabular learning is substantial but cannot be accessed by simply applying distance-based selection methods designed for vision or language to raw tabular features.
Inducing a latent geometry via unsupervised PFNs is a necessary prerequisite for effective instance selection in the cold-start setting.
Improving representation learning for tabular data is as critical as designing complex selection heuristics.

The work positions unsupervised context selection as a bridge between abundant unlabeled data and the strong, context-sensitive predictions of TFMs. The authors acknowledge limitations, including the restriction to classification tasks with $\le 10$ classes, the computational cost of K-Medoids on very large pools, and the current coupling to TabClustPFN as the representation source. They suggest future work should explore regression tasks, scalable approximations, and whether other embedding models or selection strategies can further improve performance.