📊 statistics

Efficient Benchmarking Is Just Feature Selection and Multiple Regression

This paper demonstrates that efficient LLM benchmarking can be significantly improved by reframing the problem as multiple regression with kernel ridge regression for prediction and the mRMR algorithm for selecting optimal question subsets, resulting in lower prediction errors, higher ranking correlations, and faster, more stable performance compared to existing methods.

Original authors: Sam Bowyer, Acyr Locatelli, Kris Cao

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sam Bowyer, Acyr Locatelli, Kris Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to grade a class of 50 students on a massive, 1,000-question final exam. You want to know exactly how well the class performed overall, but grading every single question for every student takes forever and costs a fortune in paper and ink.

You need a shortcut. You want to pick just a small handful of questions (say, 50 of them) that, if you grade those, will tell you almost exactly how the students would have scored on the full 1,000 questions.

This is the problem of Efficient Benchmarking for Large Language Models (LLMs). Instead of humans, we are using computers to grade AI models. The paper argues that the current ways of picking these "shortcut questions" are too complicated and often miss the mark.

Here is the paper's solution, explained simply:

1. The Old Way: Guessing and Clustering

Previous methods tried to pick questions by:

Grouping them: Like sorting a deck of cards by suit and picking one card from each suit (Clustering).
Statistical modeling: Trying to build a complex psychological profile of the questions to see which ones are "most important" (Item Response Theory).

The authors say these methods are like trying to solve a Sudoku puzzle using a supercomputer when you could just use a simple logic trick. They are slow, unstable (picking different questions every time you run them), and sometimes pick the wrong questions.

2. The New Way: "The Best Questions" + "Smart Math"

The authors reframe the problem into two simple steps, like a two-step recipe:

Step 1: The Question Selector (mRMR)
Instead of guessing, they use a method called mRMR (Minimum Redundancy, Maximum Relevance).

The Analogy: Imagine you are building a playlist to represent a whole music genre.
- Maximum Relevance: You want songs that really capture the essence of the genre (they are relevant to the whole).
- Minimum Redundancy: You don't want three songs that all sound exactly the same. If you pick one heavy metal song, you don't need two others that are identical. You want variety.
How it works: The algorithm looks at how every question relates to the final score (Relevance) and how much it overlaps with other questions (Redundancy). It greedily picks the questions that tell you the most new information without repeating what you already know.
The Result: It picks a "Core Set" of questions that are diverse and highly informative.

Step 2: The Score Predictor (Kernel Ridge Regression)
Once you have your small set of questions, you need to guess the full score based on the results of just those few.

The Old Way: Just take the average of the small set (like saying, "If they got 50% on these 50 questions, they probably got 50% on the whole test"). This is too simple.
The New Way: Use Kernel Ridge Regression.
- The Analogy: Imagine you are predicting the weather. A simple average might say, "It's 70 degrees, so it's summer." But a smart predictor knows that if it's 70 degrees and the humidity is high and the wind is blowing from the north, it might actually be a storm.
- How it works: This math technique doesn't just look at individual questions; it looks at how questions combine. It realizes that getting Question A and Question B right might be more important than just getting A right and B right separately. It finds these hidden patterns to make a much sharper prediction.

3. Why This is a Big Deal

The paper tested this against all the other fancy methods using real data from AI models. Here is what they found:

It's More Accurate: The new method made fewer mistakes when guessing the full score. Whether the test was "Right/Wrong" (Binary) or a "Score out of 100" (Continuous), the new method was closer to the truth.
It's Better at Ranking: If you want to know which AI model is the "best," this method ranks them more accurately than the others. It's less likely to say Model A is better than Model B when they are actually tied or reversed.
It's Stable: If you run the test five times, the old methods might pick five completely different sets of questions. The new method picks the same questions every time. It's reliable.
It's Fast: The old methods took a long time to compute. The new method is like a sprint compared to a marathon. It's incredibly fast, especially for binary (Right/Wrong) tests.

The Bottom Line

The authors claim that you don't need complex, heavy-duty AI models to pick the best questions for testing other AIs. You just need a smart way to pick diverse questions (mRMR) and a smart way to combine their scores (Kernel Ridge Regression).

By treating this as a simple math problem of "Feature Selection" (picking the right ingredients) and "Regression" (cooking them together), they get better results faster than anyone else. It's a reminder that sometimes, the simplest, most elegant statistical tools are the most powerful.

Technical Summary: Efficient Benchmarking Is Just Feature Selection and Multiple Regression

Problem Statement

Evaluating Large Language Models (LLMs) on comprehensive benchmarks involving hundreds or thousands of questions is computationally expensive and time-consuming, particularly as model sizes and benchmark complexity increase. Traditional full-benchmark evaluation creates bottlenecks for tasks requiring frequent evaluation, such as tracking model development during training runs or conducting ablation studies on inference-time parameters (e.g., sampling hyperparameters, prompt formats).

The goal of efficient benchmarking is to predict the full benchmark score of a test model using only a small subset (a "coreset") of the benchmark's questions. This requires a two-stage process:

Stage 1 (Selection): Selecting a subset of $n$ questions from the full dataset of $N$ questions ( $n \ll N$ ).
Stage 2 (Prediction): Learning a function to predict the full benchmark score based solely on the scores achieved on the selected coreset questions.

Existing methods often rely on clustering (e.g., K-Means) or Item Response Theory (IRT) for selection and simple weighted means or specific IRT-based predictors for estimation. The authors argue these approaches can be significantly improved by reframing the problem as a classic statistical task: multiple regression with feature selection.

Methodology

The proposed approach decouples the problem into two distinct stages, leveraging established techniques from feature selection and regression literature.

Stage 1: Feature Selection via mRMR

Instead of clustering or IRT-based difficulty estimation, the authors employ Minimum Redundancy Maximum Relevance (mRMR).

Representation: Each question $q_i$ is represented as a feature vector containing the scores of $M$ source models on that question: $q_i = [s(m_1, q_i), \dots, s(m_M, q_i)]$ .
Target: The relevance target $Y$ is the vector of summary scores (full benchmark averages) for the $M$ source models: $\bar{s} = [\bar{s}(m_1, D), \dots, \bar{s}(m_M, D)]$ .
Objective: The algorithm greedily selects questions to maximize Relevance (mutual information between the question scores and the target summary scores) while minimizing Redundancy (mutual information between the selected questions themselves).
Metric: The authors utilize the Mutual Information Quotient (MIQ) as the importance function $g$ , which balances relevance and redundancy effectively.
Estimation: For binary scores, empirical mutual information is calculated. For continuous scores, the authors use PCA-corrected Kraskov–Stögbauer–Grassberger (KSG) estimators to handle the continuous nature of the data and the discrete-continuous mix when necessary.

Stage 2: Prediction via Kernel Ridge Regression

Once a coreset $C$ is selected, the authors learn a predictor $f_C$ to map coreset scores to the full benchmark score.

Ridge Regression: A baseline linear model with L2 regularization is used.
Kernel Ridge Regression: To capture non-linear interactions between question scores (e.g., the combined difficulty of answering two specific questions), the authors employ polynomial kernel ridge regression. The kernel is defined as $\kappa(x_i, x_j) = (\langle x_i, x_j \rangle + 1)^d$ , where $d$ is the degree.
Key Finding: The authors find that setting the polynomial degree $d=2$ (quadratic kernel) consistently yields the best performance, allowing the model to consider pairwise interactions of question scores without significant overfitting.

Baselines and Comparisons

The method is compared against several existing efficient benchmarking strategies:

AnchorPoints: Uses K-Means clustering to select medoid questions and predicts via weighted mean.
gp-IRT: Uses Item Response Theory (IRT) to infer latent abilities and difficulties, then combines IRT predictions with AnchorPoints scores.
Lasso: Uses L1 regularization to perform simultaneous feature selection and prediction.
Random Search: Exhaustively samples random coresets and selects the best via cross-validation (used as a strong baseline).
MetaBench: Uses IRT and Fisher information for selection.

Key Contributions

Reframing the Problem: The paper posits that efficient benchmarking is fundamentally a feature selection and regression problem. By applying standard statistical techniques (mRMR and Kernel Ridge Regression), existing methods can be significantly outperformed.
Superior Performance of mRMR + Kernel Ridge: The combination of mRMR for coreset construction and polynomial kernel ridge regression for prediction (denoted as mRMR++) achieves state-of-the-art results across binary and continuous metrics.
Stability and Efficiency:
- Stability: Unlike IRT-based methods or clustering approaches which can vary significantly with random seeds or data splits, mRMR demonstrates high stability in the questions it selects.
- Speed: The mRMR algorithm is orders of magnitude faster than competitor methods that require fitting complex probabilistic models (like IRT) or running clustering algorithms.
Generalization: The method effectively generalizes to predicting related metrics (e.g., predicting $pass@k$ for $k > 1$ using coresets constructed from $pass@1$ data), outperforming methods that rely on specific metric assumptions.

Experimental Results

The authors evaluated their approach on a variety of datasets:

Binary Metrics: Open-LLM-Leaderboard (448 models, 7 benchmarks) and HELM-Lite (79–83 models, 6 benchmarks).
Continuous Metrics: Document summarization (GovReport, BioLaySumm) with ROUGE-L, BERTScore, and FKGL.
Pass@k Metrics: Coding benchmarks (MBPP, MBPP+, LBPP, HumanEval) with varying generation counts.

Key Findings:

Error Reduction: mRMR++ consistently achieves lower Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) compared to AnchorPoints, gp-IRT, Lasso, and random baselines.
Ranking Correlation: The method achieves higher Spearman $\rho$ and Kendall $\tau$ correlation coefficients, indicating better preservation of model rankings.
Data Regimes: The method performs exceptionally well when the number of source models ( $M$ ) is sufficient (e.g., $M \ge 30$ ). In very data-poor settings ( $M < 15$ ), the signal-to-noise ratio can degrade performance, though it often remains competitive.
Kernel Degree: A quadratic kernel ( $d=2$ ) was found to be optimal across most settings, suggesting that pairwise question interactions are the most significant non-linear feature.
Comparison to Search+: While "Search+" (exhaustive random search) is a strong baseline, mRMR++ achieves comparable or better results with a principled, deterministic selection process that is computationally cheaper.

Significance and Claims

The paper claims that the proposed approach offers a simple, fast, and highly effective alternative to complex probabilistic modeling for efficient benchmarking.

Compute Savings: By enabling accurate predictions with only 5–15% of the benchmark questions, the method offers significant computational savings, particularly for iterative tasks like training monitoring or hyperparameter tuning.
Interpretability and Robustness: The reliance on standard regression and feature selection techniques makes the method more interpretable and robust to variations in data splits compared to IRT-based approaches.
Broad Applicability: The technique is applicable to both binary (correct/incorrect) and continuous (e.g., ROUGE, pass@k) scoring metrics without requiring metric-specific adaptations.
Future Work: The authors modestly note that while mRMR works well, further improvements on continuous-valued datasets could be made by refining mutual information estimators for low-data regimes or exploring alternative feature importance functions. They also highlight that the method is best suited for in-distribution predictions and may struggle with models significantly outside the capability range of the source models used for selection.

The authors conclude that "efficient benchmarking is just feature selection and multiple regression," suggesting that researchers should leverage these well-studied statistical tools rather than developing increasingly complex, benchmark-specific heuristics.