Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths

Technical Summary: Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths

Problem Statement

Current tabular machine learning benchmarks primarily rely on aggregating performance scores (e.g., average ranks, win rates, Elo ratings) across multiple datasets to identify robust default models. While effective for selecting general-purpose baselines, the authors argue that this aggregation approach obscures a critical question: which models are necessary to attain peak performance on specific datasets?

Standard metrics conflate qualitatively different evaluation dimensions. They tend to reward models that perform consistently "decently" across all tasks while penalizing models that exhibit unique, dataset-specific strengths but may fail on others. Consequently, models that expand the state-of-the-art on specific data regimes (irreplaceable models) may appear mediocre under standard aggregation, while models that merely avoid failure are over-rewarded.

Methodology

The authors propose a data-centric peak performance frontier framework to disentangle model performance into four distinct categories based on statistical significance relative to other models on each dataset.

1. Statistical Framework:

• Indistinguishable Performance: Using paired fold-wise differences and a one-sided Wilcoxon signed-rank test ($\alpha = 0.05$), the authors determine if a model $m_i$ is significantly better than a reference model $m_j$. If the test fails to reject the null hypothesis of no improvement, the models are treated as statistically indistinguishable.
• Peak Performance Set ($P_d$): For each dataset $d$, $P_d$ contains all models that are not statistically separated from the empirical best model (the one with the lowest average error).
• Failure Definition: A model is considered to have "fallen behind" if it performs significantly worse than the median-performing model on a dataset.

2. Proposed Metrics:
Based on the position of a model relative to $P_d$ and the median, the authors define four metrics:

• Irreplaceability: The number of datasets where a model is the sole member of $P_d$. This indicates unique inductive biases that are necessary for peak performance on those specific datasets.
• Sufficiency: The number of datasets where a model belongs to $P_d$ (even if not unique). This indicates the model is a sufficient stand-alone choice for reaching peak performance.
• Redundancy: Models that do not reach the frontier ($P_d$) but are not significantly worse than the median. They are competitive but not peak-performing.
• Fallibility: The number of datasets where a model is significantly worse than the median model.

3. Experimental Setup:
The framework is applied to TabArena, a live benchmark comprising 27 models evaluated on 51 carefully curated datasets. The authors compare their frontier-oriented metrics against standard TabArena aggregation metrics (Elo, normalized error, rank, harmonic mean rank, and improvability).

Key Results

• High Correlation of Standard Metrics: Standard aggregation metrics (Elo, rank, win-rate) are highly correlated, measuring largely the same dimension: consistency and failure avoidance. They do not align well with dataset-level irreplaceability.
• Aggregation Rewards Consistency: A hypothetical model ranked consistently (e.g., 6th) across all datasets can achieve top rankings in Elo and average rank without ever being the best on a single dataset.
• Aggregation Penalizes Isolated Excellence: A hypothetical model that is first on >50% of datasets but last on the rest performs poorly under most standard metrics (except harmonic mean rank). This demonstrates that standard metrics penalize failure more strongly than they reward isolated frontier contributions.
• Hidden Strengths: Models like TabICLv2 and TabPFN-2.5 appear similar in standard rankings but are irreplaceable on distinct sets of datasets, proving they are not substitutable. Conversely, RealMLP, which ranked first in the original TabArena publication, is never irreplaceable and reaches the frontier on only two datasets; its strength lies in low fallibility (avoiding poor outcomes).
• Unique Inductive Biases: 9 out of the 27 analyzed models are irreplaceable on at least one dataset, indicating that unique inductive biases exist that are not captured by consistently strong default models.
• AutoML vs. Individual Models: While AutoML solutions (like AutoGluon) often provide robust defaults, individual models (e.g., CatBoost, TabDPT, ModernNCA) can be irreplaceable on specific datasets, revealing inductive biases that AutoML pipelines might not automatically leverage.

Significance and Claims

The paper claims that benchmark evaluation should evolve beyond selecting a single "best" default model. Instead, progress should be measured by whether new models expand the set of attainable peak performances across datasets.

• Complementary Evaluation: The proposed metrics are not intended to replace aggregate leaderboard scores but to complement them. Aggregation metrics remain useful for identifying robust defaults (low fallibility, high sufficiency), while frontier metrics identify irreplaceable contributions (unique inductive biases).
• Diagnostic Value: The framework helps distinguish between models that are merely "good on average" and those that provide non-redundant value to the model zoo. It can also expose "pseudo-improvements" where a new model copies existing performance without adding unique value.
• Practitioner Guidance: The authors suggest a staged evaluation strategy: start with low-fallibility models for reliability, move to high-sufficiency models for frontier coverage, and finally test irreplaceable models to check for specific inductive biases favorable to the target dataset.

In conclusion, the paper argues that "benchmark progress" is not solely defined by improving aggregate scores but by expanding the peak-performance frontier, ensuring that the model zoo contains diverse, irreplaceable tools for specific data regimes.