Technical Summary: VIP-COP: Context Optimization for Tabular Foundation Models

1. Problem Statement

Tabular Foundation Models (TFMs), such as TabPFN, have emerged as powerful tools for in-context learning (ICL) on structured data. These models act as "algorithms-in-a-box," performing predictions on new tasks via a single forward pass using labeled training data as context, without requiring task-specific model training. However, TFMs face a critical bottleneck: context length limits.

Pretrained TFMs are typically trained on datasets with specific size and dimensionality constraints. When inference-time data exceeds these pretraining distributions (in terms of sample count $n$ or feature count $d$), the models suffer from out-of-distribution behavior and degraded generalization. While TFMs eliminate the need for model training, they shift the optimization burden to the input context. Selecting the optimal subset of samples and features becomes a combinatorial optimization problem (Hard Context Optimization or HardCOp).

Existing approaches fall into two categories with limitations:

• Heuristics: Model-agnostic methods (e.g., random sampling, clustering) that optimize generic objectives but fail to account for the specific TFM's inductive biases.
• Soft Prompt Optimization (SoftCOp): Methods that learn abstract, continuous context tokens. These require access to model internals (gradients), produce uninterpretable latent representations, and lack the ability to explicitly filter noise or select specific data points.

The core challenge is to perform HardCOp—selecting discrete, interpretable subsets of samples and features—that maximizes performance for a frozen, black-box TFM, particularly when data is oversized, noisy, or augmented.

2. Methodology: VIP-COP

The authors propose VIP-COP (Very Important Predictors for Context OPtimization), a fast, any-time, black-box algorithm designed to identify the most influential samples and features for a given TFM task.

Core Concept: Value Attribution as Credit Allocation

VIP-COP frames context selection as a credit allocation problem. Since the performance of a TFM on a query set reflects the collective value of the context, the contribution of any single sample or feature depends on the other members of the context. The authors estimate the Value of Importance for Prediction (VIP) for each item using Shapley values from cooperative game theory.

Algorithmic Framework

VIP-COP estimates Shapley values via an extension of KernelSHAP, formulated as a linear regression problem where:

• Dependent Variable: Validation performance ($y_{val}$) obtained when a specific subset is used as context.
• Independent Variables: Binary indicators ($c_i \in \{0, 1\}$) representing the presence of a specific sample or feature in the context.
• Objective: The regression coefficients approximate the VIP scores.

To handle the combinatorial explosion of possible subsets, VIP-COP employs an iterative refinement strategy:

1. Online Regression: It performs mini-batch Stochastic Gradient Descent (SGD) to update VIP estimates.
2. Value-Guided Sampling: Instead of uniform random sampling, new context subsets are generated via importance sampling. The probability of selecting an item is proportional to its current VIP estimate, scaled by a temperature parameter $\tau$.
3. Temperature Scheduling: To balance exploration (finding new high-value items) and exploitation (refining estimates of known high-value items), the algorithm runs parallel instances with varying temperature schedules (inspired by Hyperband).
4. Multi-Fidelity Pruning: The process iterates for a user-specified budget $R$. In each round, new subsets are scored via a single TFM forward pass, and the regression is updated.

Key Properties

• Black-Box: Requires no access to model gradients or internals, making it compatible with proprietary and open-source TFMs.
• Any-Time & Budget-Aware: Performance improves monotonically with additional compute time; users can stop the optimization at any point.
• Interpretable: Outputs discrete "Very Important Predictors" (specific rows and columns), unlike soft prompts.
• Robust: Explicitly isolates signal from noise, making it effective for data augmentation (selecting useful augmented samples) and denoising (filtering irrelevant features).

3. Key Contributions

1. Test-Time Context Optimization: Introduction of VIP-COP, a method to subset-select VIP samples and features for any TFM, specifically addressing scenarios where input data exceeds context capacity.
2. Algorithmic Innovation: A novel formulation of Shapley value estimation using iterative KernelSHAP regression with value-guided importance sampling and multi-fidelity pruning. This balances rapid performance gains with accurate value estimation.
3. Desirable Properties: The method is characterized by being fast (minutes), black-box, interpretable, and robust to noise and augmentation.
4. Empirical Validation: Extensive experiments demonstrating consistent outperformance over heuristic and optimized baselines across diverse datasets, dimensions, and noise settings.

4. Experimental Results

The authors evaluated VIP-COP on 38 multi-class classification datasets from the TALENT benchmark using TabPFN-v1 (limited to $n_C=1024, d_C=100$).

• Baselines: Compared against heuristics (Random, Ensemble, XLContext) and optimized methods (KMeans Reps, Decision Tree + TFM).
• Original HardCOp: VIP-COP improved performance by 1.54% to 14.68% over baselines. It significantly outperformed all baselines except DT+TFM (which was competitive on large-$n$ datasets but less so on smaller ones).
• Data Augmentation (DA): In settings where data was augmented (potentially introducing noise), VIP-COP improved performance by 4.80% to 15.05%. It successfully exploited informative augmentations while filtering out uninformative ones, a capability where other baselines struggled.
• Data Noise (DN): Under noisy conditions (dropped samples/features replaced with noise), VIP-COP showed massive gains, particularly in feature noise scenarios (154.11% to 240.81% improvement), demonstrating superior robustness in isolating signal.
• Efficiency: The algorithm runs at approximately 1.4 seconds per round. Performance gains are typically achieved within a few minutes (e.g., ~2.5 minutes for 100 rounds), establishing a favorable performance-latency trade-off (Pareto frontier).

5. Significance and Claims

The paper claims that VIP-COP elevates context optimization from a heuristic practice to a core, principled component of Tabular Foundation Models. By treating context selection as a value attribution problem, VIP-COP enables TFMs to:

• Operate effectively on data scales exceeding their pretraining limits.
• Leverage data augmentation strategies without being degraded by the associated noise.
• Function as robust, black-box tools for both proprietary and open-source models.

The authors position VIP-COP as a solution that unifies effectiveness, efficiency, and robustness, allowing for reliable in-context learning on structured data without requiring architectural changes to the underlying foundation model. The work suggests that for TFMs, the "model" is effectively the context, and optimizing this context is the key to unlocking their full potential in real-world, noisy, and large-scale scenarios.