Technical Summary: LimiX-2M

1. Problem Statement

Tabular Foundation Models (TFMs) have recently demonstrated the ability to rival or surpass traditional tree ensembles (e.g., XGBoost, CatBoost). However, the authors identify two critical inefficiencies in current TFM architectures that limit their accuracy-efficiency trade-off:

1. Low-Rank Collapse in Embeddings: Standard TFMs (e.g., TabPFN-v2) typically map scalar numerical cells to high-dimensional hidden spaces via a single linear projection, augmented by feature IDs or positional embeddings. The authors argue that this "affine scalar tokenization" injects value variation through an essentially one-dimensional channel. Consequently, feature matrices in shallow layers exhibit extremely low effective rank (sometimes collapsing to single-digit ranks), leading to significant parameter redundancy and weak value sensitivity. Theoretical analysis (Proposition 3.1 and Theorem B.1) suggests that without additional nonlinearity, the rank of embedded inputs and early hidden states is bounded by a small constant (e.g., 2 for single-head attention, $H+1$ for multi-head), regardless of the hidden dimension size.
2. Attention Bottlenecks and Misalignment: Existing bidirectional attention stacks often follow a Feature-Attention $\to$ Sample-Attention $\to$ FFN (F$\to$S$\to$N) order. This ordering forces feature-level attention to integrate across columns based solely on raw, weakly conditioned values before any column-level statistics are established. Furthermore, since final predictions often consume only the target token, the sample-level attention computations performed over features are effectively ignored, leading to weak training signals for parts of the network that do not directly influence the readout.

2. Methodology

The paper proposes LimiX-2M, a unified framework addressing these issues through two primary components:

A. RaBEL (Radial-Basis Embedding Layer)

To mitigate low-rank collapse, the authors replace the standard linear projection with RaBEL, a compact, localized Radial Basis Function (RBF) expansion.

• Mechanism: Instead of a direct linear projection, each scalar input $x_{i,j}$ is expanded into a bank of localized nonlinear features $\phi_j(x_{i,j})$ using Gaussian kernels. These features are then projected to the model dimension.
• Exponent Gating: To handle real-world tabular data that spans orders of magnitude and exhibits heteroskedasticity, RaBEL introduces an "exponent gate." This mechanism conditions the RBF centers and bandwidths based on the log-magnitude of the input value. This allows the model to maintain scale invariance and locality across different value regimes without brittle hard binning.
• Effect: This design injects nonlinearity at the tokenizer stage, increasing the effective rank of shallow representations and improving the conditioning of the first learned layer without requiring deep stacking to discover curvature.

B. Reordered Bidirectional Attention (S$\to$N$\to$F)

The authors propose reordering the attention stack from the standard F$\to$S$\to$N to Sample-Attention $\to$ FFN $\to$ Feature-Attention (S$\to$N$\to$F).

• Sample-Attention First: The model first aggregates column-level correlations and distributional statistics (e.g., moments, prevalence) across samples.
• Intermediate Conditioning: A lightweight Feed-Forward Network (FFN) compresses and conditions these signals.
• Feature-Attention Second: The feature-attention layer then models inter-feature relations using these richer, better-conditioned inputs.
• Readout Alignment: The final prediction is obtained via attention pooling over all feature tokens, ensuring that all attention computations (both sample and feature levels) contribute directly to the output, thereby strengthening training signals.

3. Key Contributions

1. Diagnosis of Low-Rank Collapse: The paper provides theoretical proofs and empirical evidence (via Singular Value Decomposition of hidden states) demonstrating that standard linear+ID embeddings induce severe low-rank collapse in early layers, rendering large hidden dimensions inefficient.
2. RaBEL Architecture: The introduction of a compact, RBF-based cell encoder that expands scalar inputs into localized nonlinear features, significantly boosting the effective rank of shallow representations and improving conditioning.
3. Reordered Attention Stack: The identification of a "permutation-order pathology" in standard bidirectional attention and the proposal of an S$\to$N$\to$F stack that establishes column-level statistics before feature aggregation and routes all attention signals to the readout.
4. LimiX-2M Model: The realization of a 2M-parameter model that integrates these components, demonstrating that principled nonlinear embeddings and attention-order redesign can unlock better accuracy-efficiency trade-offs.

4. Experimental Results

The authors evaluated LimiX-2M against a wide range of baselines, including tree ensembles (XGBoost, LightGBM, CatBoost), specialized deep architectures (TabNet, SAINT, FT-Transformer), and other foundation models (TabPFN-v2, TabICL, Mitra).

• Performance: Despite having only 1.92M parameters, LimiX-2M outperforms the 7.24M-parameter TabPFN-v2 and the 27M-parameter TabICL on most widely used tabular benchmarks (covering classification and regression tasks across OpenML-CC18, TabZilla, TabArena, TALENT, and BCCO suites).
• Efficiency: The model achieves significantly lower training and inference costs. On GPU, LimiX-2M is approximately 2$\times$ faster than TabPFN-v2 and >10$\times$ faster than TabICL.
• Ablation Studies:
  • Replacing the linear embedding with RaBEL significantly increases the numerical rank and Rank@99% of the first three layers compared to a baseline with the same architecture.
  • The S$\to$N$\to$F ordering (Reordered Bidirectional Attention) consistently outperforms the standard F$\to$S$\to$N ordering.
  • Toy experiments on synthetic Directed Acyclic Graphs (DAGs) show that the S$\to$N$\to$F architecture assigns higher attention scores to direct causes of the target variable compared to the standard ordering.

5. Significance and Claims

The paper claims that the primary limitations of current TFMs are not merely a lack of capacity, but structural inefficiencies in how values are tokenized and how attention is routed. By addressing the "value bottleneck" via RaBEL and the "readout misalignment" via reordered attention, LimiX-2M demonstrates that:

• Value-aware tokenization is a key lever for improving the accuracy–efficiency trade-off.
• Readout-aligned routing ensures that all computational resources contribute to the final prediction.
• High performance on tabular data does not necessarily require massive parameter counts; a 2M-parameter model can surpass larger foundation models when the architecture is optimized for the specific statistical properties of tabular data (e.g., piecewise trends, heavy tails, and mixed types).

The authors conclude that these findings suggest a path toward more reliable scaling for tabular foundation models, moving beyond simple parameter scaling to architectural principles that better align with the data's intrinsic structure.