Technical Summary: TFM-Retouche

Problem Statement

Tabular Foundation Models (TFMs), such as TabPFN-2.6 and TabICLv2, have demonstrated strong zero-shot performance through In-Context Learning (ICL) by leveraging transferable inductive biases learned from synthetic pretraining. However, these models possess fixed inductive biases at inference time. Adapting them to specific downstream datasets typically requires either full fine-tuning, which is computationally prohibitive, or Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA. PEFT approaches are often architecture-specific, requiring manual insertion of trainable modules into the backbone, and recent empirical evidence suggests that weight-space adaptation can sometimes degrade accuracy or calibration by eroding the fragile synthetic priors encoded in the pretrained weights. Furthermore, existing input-space adaptation methods (e.g., BETA) often rely on fixed-width bottlenecks designed for earlier, capacity-constrained models, which may not align with the alignment needs of modern, high-capacity TFMs.

Methodology

The authors propose TFM-Retouche, a lightweight, architecture-agnostic input-space residual adapter designed to align frozen TFM backbones with specific downstream tasks without modifying the pretrained weights.

Core Architecture

The framework introduces a gated residual adapter, $g_\phi(x)$, applied to the input features $x$ before they enter the frozen TFM $f_\theta$:
$$g_\phi(x) = (1 - \alpha) \odot x + \alpha \odot \delta(x)$$
Where:

• $\alpha$: A learnable gate parameter (either a scalar or a per-channel vector) that controls the strength of the intervention. It is initialized to a small value (0.02) to ensure the adapter starts near the identity function.
• $\delta(x)$: A trainable inner block transformation that maps $R^d \to R^d$. The authors implement two variants:
  1. Cross Block (DCNv2): A cross-network that explicitly models multiplicative feature interactions anchored to the original input.
  2. Residual MLP Block: A standard residual bottleneck MLP.
• Dimension Preservation: Unlike prior encoders that compress inputs, TFM-Retouche preserves the input dimensionality, acting as a "nudge" rather than a replacement.

Training and Safety Mechanisms

• End-to-End Training: The adapter parameters are trained via backpropagation through the frozen TFM backbone. The backbone weights remain completely unchanged.
• Identity Guard: To mitigate the risk of adaptation degrading performance, a post-training identity guard is employed. During inference, the system evaluates the adapter's performance on a held-out validation set. If the adapter does not improve the baseline performance by a small tolerance (default 0.5%), the system bypasses the adapter entirely, falling back to the unmodified TFM. This ensures the adaptation is strictly opt-in per dataset.

Key Contributions

1. TFM-Retouche Framework: A novel, dimension-preserving input-space adapter that is agnostic to the TFM backbone architecture. It supports both classification and regression and can be instantiated with modular inner blocks (DCNv2 or MLP).
2. Training Strategy with Identity Guard: A method for training adapters end-to-end through frozen TFMs, coupled with a safety mechanism that prevents performance degradation by reverting to the zero-shot baseline when adaptation is unhelpful.
3. Comprehensive Evaluation: The framework was instantiated on TabICLv2 (denoted as TabICLv2-Retouche) and evaluated on TabArena-Lite, a benchmark comprising 51 datasets across binary classification, multiclass classification, and regression.
4. Open-Source Release: The authors commit to releasing the framework as an open-source library.

Experimental Results

The evaluation was conducted on the TabArena-Lite leaderboard (51 datasets). Key findings include:

• Performance: TabICLv2-Retouche (T+E variant, tuned and ensembled) achieved the top rank (Elo 1651) on the leaderboard, outperforming the frozen TabICLv2 base (+56 Elo) and other state-of-the-art methods like TabPFN-2.6 and RealTabPFN-2.5.
• Efficiency: The method achieved this performance with a significantly smaller hyperparameter optimization (HPO) budget (10 random configurations vs. the standard 200 used by other tuned peers).
• Compute Trade-offs:
  • Training Time: TabICLv2-Retouche (T+E) sits on the Pareto frontier for training time versus quality, dominating other tuned-and-ensembled methods (e.g., RealMLP, RealTabPFN-2.5) which require significantly more training time.
  • Inference Time: It remains on the inference-time Pareto frontier, offering high-quality predictions with moderate inference latency compared to the fastest zero-shot models.
• Ablation Studies:
  • The Identity Guard was identified as the most critical component; removing it caused the trained adapter to perform worse than the unmodified base on average.
  • Weight-space adaptation (LoRA and full SFT) applied to the frozen TabICLv2 backbone failed to improve performance over the zero-shot baseline in this setting, whereas the input-space residual nudge succeeded.
  • BETA-style encoders (fixed-width bottlenecks) performed poorly on TabICLv2, confirming that the bottleneck for modern TFMs is alignment, not capacity.

Significance and Claims

The paper positions TFM-Retouche not as a replacement for zero-shot TFM workflows, but as a low-risk, late-stage refinement mechanism.

• Alignment over Capacity: The work argues that for modern, high-capacity TFMs, the primary challenge is aligning the generic synthetic prior with specific task structures, rather than compressing input capacity.
• Safety and Modularity: By leaving the backbone frozen and using an identity guard, the framework allows practitioners to capture marginal predictive gains without the risk of destabilizing the model or committing to a specific backbone architecture.
• Practicality: The method requires minimal computational resources (training only 102–105 parameters per dataset) and is compatible with existing zero-shot pipelines, making it a practical tool for improving performance once upstream data decisions have stabilized.

The authors conclude that input-space adaptation can deliver state-of-the-art quality without sacrificing efficiency, provided the adaptation is lightweight, near-identity, and guarded against performance degradation.