Technical Summary: LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning

Problem Statement

Recent benchmarks in tabular deep learning reveal a "tight performance cluster" where leading methods—including gradient-boosted decision trees (GBDTs), attention-based architectures, and implicit ensembles like TabM—perform with comparable efficacy. While deep ensembles are a proven method for improving model performance, training $K$ independent networks scales linearly with cost. Implicit ensembles, such as BatchEnsemble and TabM, address this by sharing most parameters while introducing member-specific perturbations.

However, existing implicit ensembles rely on a fixed rank-1 structure. In TabM, each member modulates a shared weight matrix $W$ via a rank-1 mask ($W_k = W \odot (s_k r_k^\top)$). This design lacks an explicit axis for increasing the capacity of member-specific modulation. Consequently, ensemble diversity—the key driver of ensemble gains—is not directly controllable; it must emerge implicitly from fixed rank-1 masks and training dynamics. Prior work suggests that under certain conditions, BatchEnsemble members can converge to near-identical functions, indicating that rank-1 modulation may fail to realize meaningful diversity.

Methodology: LoMETab

The authors propose LoMETab, a rank-$r$ generalization of multiplicative implicit ensembles designed to overcome the limitations of fixed rank-1 structures.

Core Formulation

Instead of a rank-1 mask, LoMETab assigns each member $k$ a pair of low-rank adapter matrices, $A_k \in \mathbb{R}^{d_{out} \times r}$ and $B_k \in \mathbb{R}^{d_{in} \times r}$. The effective weight for member $k$ is defined as:
$$W_k = W \odot (1 + A_k B_k^\top)$$
Here, $W$ is the shared base weight, and $A_k B_k^\top$ forms a residual matrix of rank at most $r$. The term $(1 + A_k B_k^\top)$ acts as an identity-residual multiplicative perturbation. When the residual is near zero, the member behaves like the shared weight; larger residuals induce member-specific deviations.

Architecture and Training

• Architecture: LoMETab is inserted into a tabular MLP backbone. A shared piecewise linear embedding processes the input, which is then copied across $K$ members. Each hidden linear layer is replaced by a LoMETab Linear layer, combining the shared weight with member-specific adapters.
• Training Objective: The model is trained using a member-wise loss function, optimizing the average task loss across all $K$ members. This allows parallel submodels to specialize despite sharing the backbone.
• Inference: Final predictions are obtained by averaging the outputs of all $K$ members (averaging probabilities for classification, scalar outputs for regression).

Theoretical Expressivity

The paper establishes a strict hierarchy in hypothesis classes. For any rank $r \ge 2$, the set of representable effective weights in LoMETab strictly contains the set representable by BatchEnsemble (rank-1). Specifically, any BatchEnsemble mask $s_k r_k^\top$ can be represented as $1 + A_k B_k^\top$ because the difference $s_k r_k^\top - 1$ has a rank of at most 2. Thus, LoMETab strictly enlarges the design space for $r \ge 2$.

Key Contributions

1. Rank-$r$ Generalization: Introduction of LoMETab, which replaces fixed rank-1 masks with identity-residual low-rank Hadamard perturbations, parameterized by adapter rank $r$ and initialization scale $\sigma_{init}$.
2. Controllable Diversity: Demonstration that the added rank capacity manifests as measurable predictive diversity. The authors show that LoMETab sustains higher pairwise Kullback–Leibler (KL) divergence than additive low-rank ablations. Furthermore, the $(r, \sigma_{init})$ grid provides broad control over diversity, varying pairwise KL by several orders of magnitude.
3. Dataset-Dependent Tuning: Empirical evidence that optimal configurations for rank and initialization scale are dataset-dependent. The paper argues against a "fixed rank-1" construction, proposing instead a controllable family of implicit ensembles where $r$ and $\sigma_{init}$ serve as practical tuning axes.

Experimental Results

The authors evaluated LoMETab on 37 academic datasets (including the TabM benchmark and TabZilla sources) and 77 evaluation tasks.

• Benchmark Performance: LoMETab achieves a mean ranking of 2.1 ± 1.3 across 37 datasets, outperforming TabM (2.3 ± 1.8) and competing with strong GBDT baselines (e.g., XGBoost, CatBoost) and recent neural architectures. It ranks first on 14 datasets and within the top three on 89.2% of datasets.
• Diversity Control:
  • Multiplicative vs. Additive: LoMETab maintains a "sustained gap" in pairwise KL divergence compared to an additive low-rank ensemble ($W_k = W + A_k B_k^\top$), which rapidly converges to low diversity.
  • Grid Sweep: Sweeping $r \in \{1, 2, 4, 8, 16\}$ and $\sigma_{init} \in \{0.1, \dots, 3.0\}$ reveals that diversity is jointly controlled by these axes. Low $\sigma_{init}$ results in low diversity regardless of rank, while higher $\sigma_{init}$ activates the rank axis, significantly increasing diversity.
  • Task-Level Metrics: Increased diversity correlates with higher argmax disagreement in classification and higher ambiguity in regression.
• Performance vs. Diversity Trade-off: While diversity generally improves performance, the relationship is not strictly monotonic. Tuning $r$ and $\sigma_{init}$ allows for dataset-specific optimization, often outperforming simple scaling of the ensemble size $K$ in TabM.
• Parameter Efficiency: Despite the increased expressivity, LoMETab does not necessarily incur a higher parameter count than TabM. Hyperparameter optimization often selects more compact configurations (e.g., smaller hidden dimensions) when using higher ranks, leading to comparable or even lower total parameter counts in many cases.

Significance and Claims

The paper claims that for tabular implicit ensembles, how members are allowed to deviate from a shared backbone is a central design question. By moving beyond the fixed rank-1 constraint, LoMETab exposes the adapter rank and initialization scale as practical, controllable axes for managing ensemble diversity.

The authors modestly frame their contribution as providing a "controllable family of implicit ensembles" rather than a fixed solution. They emphasize that the optimal configuration is dataset-dependent and that the primary value of LoMETab lies in its ability to tune the form and scale of member deviations to suit specific tasks. The work suggests that future research could explore whether this controllability translates to improved uncertainty estimation or out-of-distribution detection, though these are identified as directions for future work rather than current results.