📊 statistics

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

This paper introduces a framework for selecting representative dataset subsets to efficiently benchmark machine learning models while preserving global rankings, demonstrating that strategies like farthest-first selection can achieve high correlation with full benchmarks in time series classification but showing limited effectiveness in recommender systems.

Original authors: Rostislav Gusev, Alexey Zaytsev

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rostislav Gusev, Alexey Zaytsev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a food critic trying to decide which of 100 different restaurants serves the best pizza. You have a limited budget and time, so you can't visit every single one. You want to pick just a few "representative" restaurants to visit, hoping that the ranking you create from those few visits will match the ranking you would have gotten if you had visited all 100.

This paper is about solving that exact problem, but for Artificial Intelligence (AI) models instead of pizza.

The Problem: The "Pizza Review" Dilemma

In the world of AI, researchers constantly build new models to solve problems (like predicting stock prices or recognizing handwritten notes). To see which model is the "best," they test them on huge collections of datasets (like 100 different pizza recipes).

However, testing a model on 100 datasets takes forever and costs a lot of money. So, people often just pick a small handful of datasets (say, 5 or 10) to test on. The problem is: How do you pick those 5 or 10?

If you pick them randomly, you might accidentally pick only "easy" datasets, making a mediocre model look like a genius.
If you pick them based on a hunch, you might miss the datasets that actually show the difference between a good model and a great one.

The authors ask: Can we pick a tiny, smart subset of datasets that gives us the same "winner" as testing on the whole massive collection?

The Solution: The "Smart Sampler" Framework

The authors built a new system (a framework) to test different ways of picking these small subsets. They treat the datasets like points on a map. The goal is to pick points that are spread out enough to cover the whole map, so you don't miss any "territory."

They tested four main strategies for picking these points:

The Random Picker: Just grabbing datasets by chance (the baseline).
The Clusterer (K-Means): Grouping similar datasets together and picking one "representative" from each group.
The "Farthest-First" Traveler (FAFI): Starting with one dataset, then picking the next one that is as far away as possible from the first, then the next one farthest from those two, and so on. This ensures maximum diversity.
The Statistician (A/D-optimality): Using complex math to pick datasets that reduce uncertainty the most.

The Results: It Depends on the "Map"

The researchers tested this on three different worlds: Time Series (predicting trends over time), Recommender Systems (like Netflix suggesting movies), and Natural Language Processing (understanding human text).

Here is what they found, using simple analogies:

Time Series (The Clear Winner):
In this world, the "map" of the datasets was very clear. When they used the "Farthest-First" strategy (picking the most different datasets), they could pick just 5 datasets out of 112 and still get a ranking of AI models that was 95% identical to the ranking from testing all 112. It was like picking 5 diverse pizza slices and perfectly guessing the order of all 100 restaurants.
Natural Language (The Runner-Up):
Similar to Time Series, if they used smart descriptions (like summarizing the dataset with a sentence and turning that into a map), the "Farthest-First" strategy worked very well. They could save a lot of time while keeping the rankings accurate.
Recommender Systems (The Tricky One):
Here, the "map" was blurry. The features they had to describe the datasets (like how many users or items were in the database) didn't seem to capture what actually made the AI models different. In this case, picking smartly didn't help much. The "Farthest-First" strategy performed almost the same as just picking randomly. It's like trying to judge the best pizza place by looking only at the size of the parking lot; the size doesn't tell you about the taste, so picking based on size doesn't help you find the best food.

The "Secret Sauce": Good Descriptions Matter

The paper makes a crucial point: The strategy only works if you have a good way to describe the datasets.

They ran a "synthetic" experiment where they created a fake world.

When they gave the AI a "perfect description" of the datasets, the smart picking strategy worked wonders.
When they gave the AI a "broken description" (full of noise and irrelevant info), the smart strategy failed and was no better than random guessing.

The Takeaway

This paper provides a rulebook for researchers who want to save time.

Don't just guess: Use a systematic method to pick your test datasets.
Use the "Farthest-First" method: It's simple and often the best at finding diverse datasets.
Check your descriptions first: If your way of describing the datasets (the "meta-features") is good, you can cut your testing time by 90% and still know who the winner is. If your descriptions are weak, cutting corners won't help; you might as well test on everything or find better descriptions.

In short: You can eat a smaller slice of the benchmark pie and still taste the whole meal, but only if you know how to pick the right slices.

Technical Summary: Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

Problem Definition

The paper addresses the growing resource intensity of multi-dataset benchmarks in machine learning. As the number of datasets and model architectures (including deep ensembles) increases, exhaustive evaluation becomes prohibitively expensive. Practitioners often resort to evaluating models on smaller, manually selected subsets of datasets. However, this practice introduces two critical issues:

Statistical Instability: Conclusions drawn from small subsets may not reliably reflect overall superiority.
Selection Bias: The apparent "leader" in a benchmark may be an artifact of the specific tasks chosen rather than a genuine improvement, a phenomenon related to the "benchmark lottery" effect.

The core problem is formalized as dataset subset selection for benchmarking: selecting a small subset of $k$ datasets ( $k \ll n$ ) from a full pool of $n$ datasets such that the global model ranking induced by the subset ( $R_S$ ) faithfully preserves the ranking induced by the full benchmark ( $R_D$ ). The selection must rely solely on a priori dataset representations (meta-features, embeddings) without access to the full model evaluation results.

Methodology

1. Evaluation Protocol

The authors propose a unified evaluation protocol to assess selection strategies robustly:

Bootstrap Aggregation: The protocol generates $T$ independent evaluation trials by randomly perturbing the dataset pool (sampling a fraction $\alpha$ of datasets) and/or the model pool.
Ranking Construction: For each trial and subset size $k$ , a selection strategy chooses a subset $S$ . Models are ranked on $S$ and compared against the reference ranking derived from the full benchmark (or the full benchmark restricted to the sampled models).
Metrics: Performance is measured using rank-based discrepancy metrics: Mean Absolute Error (MAE) of ranks, Spearman's $\rho$ , Kendall's $\tau$ , NDCG@5, and Mean Reciprocal Rank (MRR).
Aggregation: Results are aggregated over subset sizes $k \in [2, 20]$ using the Area Under the Curve (AUC) of the performance-vs.-size curves, accompanied by 95% confidence intervals.

2. Selection Strategies

The paper evaluates four families of strategies, all operating on dataset feature vectors $x \in \mathbb{R}^p$ :

Random Baseline: Uniform sampling without replacement.
Clustering (K-Means): Selecting the point closest to the centroid of each of $k$ clusters.
Greedy Farthest-First (FAFI): Iteratively selecting the dataset that maximizes the distance to the nearest already-selected dataset. The authors test both Cosine and Euclidean distances.
Optimal Experimental Design (A-/D-optimality): Greedy selection based on minimizing the trace of the inverse Fisher information matrix (A-optimality) or maximizing the determinant (D-optimality), treating the selection as a linear regression design problem.

3. Dataset Representations

The effectiveness of selection depends heavily on the quality of the dataset embedding:

Time Series Classification (TSC): Uses simple statistics, TSFRESH, Catch22, Summary, MiniRocket embeddings, and "Landmarking" (performance of cheap probe learners like decision stumps and 1-NN).
Recommender Systems (RecSys): Uses user-item interaction matrix statistics (size, sparsity, Gini index, popularity bias).
Natural Language Processing (NLP): Uses BERT-embedded structured task descriptions and simple structured metadata from MTEB.

4. Theoretical Analysis

For the Farthest-First (FAFI) strategy, the authors derive theoretical bounds linking geometric coverage in the representation space to ranking accuracy:

MSE Decay: They model model performance as a Gaussian Process (GP). They prove that FAFI selection yields an upper bound on the Mean Squared Error of performance predictions that decays as a function of the subset size $t$ and the covering radius of the selection.
Ranking Error Bound: They derive a bound on the expected fraction of sign errors (ranking inversions) between two models. The bound shows that as the subset size grows and the posterior variance contracts (due to better geometric coverage), the probability of ranking errors decreases. This provides a theoretical justification for why FAFI works when the representation space geometry correlates with model performance differences.

Key Results

The study was conducted across three domains: Time Series Classification (112 datasets), Recommender Systems (30 datasets), and NLP (57 tasks from MTEB).

1. Domain-Specific Performance

Time Series Classification (TSC): Selection strategies significantly outperform random sampling. The best-performing strategy (Euclidean FAFI with MiniRocket embeddings) achieved a Spearman correlation of 0.95 with the full benchmark using only 5 datasets. Diversity-based methods (FAFI) and clustering generally outperformed random baselines when the representation captured ranking-relevant structure.
Natural Language Processing (NLP): Similar to TSC, geometry-based selection (specifically Cosine FAFI with BERT-embedded task descriptions) substantially outperformed random sampling, particularly for small subset sizes. This suggests that semantic descriptions of tasks provide a useful geometry for benchmark construction.
Recommender Systems (RecSys): Improvements over random selection were small and typically statistically insignificant. While K-Means showed a slight improvement in MAE, no method consistently preserved rank order better than random sampling. The authors conclude that standard dataset priors in RecSys fail to capture the factors driving model differentiation.

2. The Role of Representation Quality

A synthetic ablation study ("Oracle" vs. "Broken" representations) confirmed that the success of geometric selection is contingent on the alignment between the dataset representation and the true ranking structure:

When the representation closely reflected the latent geometry of model performance (Oracle), FAFI significantly reduced ranking errors.
When the representation was dominated by noise (Broken), FAFI provided no advantage over random sampling.

3. Practical Guidance

Subset Size: For TSC and NLP, small subsets ( $k \approx 5$ ) can achieve high rank correlation ( $\rho \ge 0.90$ ) with the full benchmark. However, stricter targets (e.g., low MAE) or conservative confidence intervals require larger subsets ( $k \approx 15-20$ ).
Statistical Significance: In TSC and NLP, the gains of non-random selection over random baselines are highly significant ( $p < 0.01$ ). In RecSys, the gains are not statistically significant.

Significance and Claims

The paper claims to provide a unified framework for treating dataset subset selection as a measurable, rank-preservation problem rather than a heuristic convenience. Its primary contributions are:

Protocol: A rigorous evaluation protocol using bootstrap aggregation to provide valid confidence intervals for comparing selection strategies across different subset sizes and domains.
Theory: The first theoretical bounds linking geometric coverage (via Farthest-First selection) to the decay of ranking errors in benchmark settings, moving beyond simple coverage metrics to statistical accuracy guarantees.
Empirical Insight: The demonstration that the efficacy of dataset selection is not universal but domain-dependent and representation-dependent. While effective for TSC and NLP where rich dataset descriptors exist, it offers little benefit for RecSys with current feature sets.

The authors emphasize that their work does not propose a new benchmark but rather a method to construct compact benchmarks that preserve the conclusions of full-scale evaluations, provided that high-quality dataset representations are available. The code and protocols are made public to facilitate reproducible, efficient benchmarking.