Cost-Efficient Estimation of General Abilities Across Benchmarks

1. Problem Statement

The evaluation of Large Language Models (LLMs) has become fragmented, with thousands of diverse benchmarks measuring performance across various domains. However, research suggests that LLM capabilities are low-rank, meaning performance across many tasks is driven by a small set of latent "abilities." While this implies that benchmarking could be compressed, current methods face two major challenges:

1. Lack of Predictive Validity: It is difficult to determine if a compressed benchmark is "good" because there is no clear metric for ability-oriented evaluation. Success is often measured by how well a subset predicts performance on the full set, but this is circular if the full set is the ground truth.
2. Variable Evaluation Costs: Existing efficient evaluation methods often treat all items as having equal cost. In reality, the token cost (and thus API cost/compute time) of items varies drastically across datasets (e.g., a complex coding task vs. a simple multiple-choice question). A method that selects 100 cheap items might be more cost-effective than one selecting 10 expensive items, even if the latter has higher information gain per item.

The core problem addressed is: How can we efficiently estimate an LLM's general abilities across a vast, unseen set of tasks while minimizing the total token cost?

2. Methodology

The authors propose a framework grounded in Predictive Validity, where the quality of an evaluation method is defined by its ability to predict model performance on unseen tasks.

A. The WILD Dataset

To enable large-scale analysis, the authors constructed the Wide-scale Item Level Dataset (WILD):

• Scale: 109,564 unique items across 163 tasks drawn from 27 different benchmarks.
• Coverage: Evaluations of 65 distinct LLMs (ranging from 0.5B to large closed API models).
• Structure: Every model-task pair is evaluated, creating a dense matrix of item-model response pairs.
• Key Feature: Unlike previous datasets, WILD captures the varying token costs of items across different domains.

B. The Nested Multidimensional IRT (MIRT) Model

The authors adapt psychometric models to estimate latent abilities:

• Multidimensional IRT (MIRT): Instead of a single scalar ability ($\theta$), models are represented by a vector of abilities ($\Theta_m \in \mathbb{R}^d$). Items have loading vectors ($K_x$) that weight these dimensions.
• Nested Structure: They modify the standard bifactor model to include a general factor ($\Theta_{m,g}$) active for all items, plus task-specific latent factors. This ensures a shared "general intelligence" component while capturing specific task nuances.
• Regularization: A regularized log-likelihood encourages shrinkage toward a unidimensional model to improve stability in low-sample regimes.
• Fitting: Parameters are learned via coordinate gradient descent. For new models, abilities are estimated using Maximum A Posteriori (MAP) estimation with an empirical prior derived from the training set.

C. Optimal Experimental Design (OED) for Item Selection

To select the most informative items, the authors move beyond simple Fisher Information (which is ill-defined for matrices in MIRT) to V-Optimality:

• V-Optimality: Minimizes the prediction variance of observed points (target benchmarks). It prioritizes items whose loadings align with the "general" factors of the target tasks.
• Adaptive Selection: Items are selected iteratively. After each response, the posterior covariance of the model's ability is updated (using the Woodbury matrix identity for efficiency), and the next item is chosen to maximize information gain.

D. Cost-Aware Selection (The $\tau$-Suffix)

To address the token cost disparity, the authors introduce a cost-aware discount factor:

• Metric: $\tau_x$ represents the average token cost of an item $x$ across training models.
• Selection Criterion: The selection score is modified to be the ratio of Information Gain / Expected Cost ($I_x / \tau_x$).
• Goal: This biases the selection toward items that offer high predictive value per token, rather than just high predictive value.

3. Key Contributions

1. WILD Dataset: Release of the largest cross-task item-level dataset (109k items, 65 models, 163 tasks) enabling the first rigorous analysis of cross-task prediction under budget constraints.
2. Predictive Validity Framework: A novel evaluation paradigm that assesses benchmarking methods based on their ability to predict performance on 112 held-out tasks (extrapolation), rather than just compressing a single known benchmark.
3. Nested MIRT + OED: Demonstration that a modified MIRT model combined with V-Optimal selection significantly outperforms unidimensional IRT and linear regression baselines in extrapolation tasks.
4. Cost-Aware Selection: Introduction of a token-cost-aware selection criterion that reduces evaluation costs by 85% while maintaining prediction accuracy.

4. Results

The experiments evaluate performance using Mean Absolute Error (MAE) on held-out tasks and Total Token Count.

• Prediction Accuracy (Extrapolation):

  • The Nested MIRT with V-Optimal selection achieves a Mean Absolute Error (MAE) of <7% on 112 held-out tasks after observing only 16 items.
  • This significantly outperforms unidimensional IRT (2PL) and simple baselines (Linear Regression, Sample Mean), especially as the number of calibration tasks increases.
  • The model successfully generalizes latent capability representations from seen tasks to unseen ones.

• Cost Efficiency:

  • Token Reduction: By incorporating cost-aware discounting, the method reaches the 7% MAE threshold using only 22,000 tokens, compared to 141,000 tokens for random selection.
  • Efficiency Gain: This represents an 85% reduction in total evaluation cost.
  • Selection Behavior: The cost-aware selector avoids expensive, low-yield items (e.g., complex reasoning tasks with high token counts) and prioritizes cheaper, high-information items (e.g., specific MMLU-Pro subtasks), whereas non-cost-aware methods often waste tokens on expensive items that do not proportionally improve prediction.

• Adaptive vs. Static:

  • Adaptive selection (iteratively picking items based on previous responses) consistently outperforms static selection (picking a fixed set of items upfront), particularly for the MIRT model.

5. Significance

• Practical Benchmarking: This work provides a principled, cost-effective method for evaluating LLMs. Instead of running models on massive, expensive benchmarks, practitioners can use a small, adaptively selected subset of items to accurately estimate general model abilities.
• Psychometrics in NLP: It successfully bridges the gap between classical psychometrics (IRT, OED) and modern LLM evaluation, proving that these statistical frameworks are robust enough for the scale and complexity of modern AI models.
• Economic Impact: By reducing evaluation costs by 85%, this approach makes rigorous benchmarking accessible for organizations with limited compute budgets and accelerates the iteration cycle for model development.
• Generalizability: The framework is designed to handle "unseen" tasks, making it a viable tool for predicting how models will perform on emerging capabilities or new benchmarks without re-evaluating the entire dataset.

In summary, the paper establishes that predictive validity is the gold standard for benchmark quality and demonstrates that cost-aware, adaptive MIRT is the most efficient method to achieve high-accuracy ability estimation across diverse LLM tasks.