Technical Summary: Efficient Evaluation of LLM Performance with Statistical Guarantees

Problem Statement

The rapid proliferation of Large Language Models (LLMs), with hundreds of thousands of variants available on platforms like HuggingFace, has created a bottleneck in evaluation. Exhaustively testing candidate models on large, curated benchmark suites (e.g., MMLU-Pro with 12,000 questions) is prohibitively expensive in terms of API costs, runtime, and expert annotation. While organizations often rely on "vibes"-based judgments on ad-hoc subsets, this lacks statistical rigor.

The paper frames this challenge as a finite-population inference problem. Given a fixed bank of $N_q$ questions and a new model, the goal is to estimate the model's accuracy $\theta$ (the proportion of correct answers over the entire bank) using a limited query budget $n_b$. The critical constraint is that the resulting confidence interval (CI) must maintain valid frequentist coverage (e.g., 95% of the time, the true $\theta$ lies within the reported interval). This guarantee is essential for reliable model selection and monitoring, particularly when adaptive evaluation strategies might otherwise invalidate standard i.i.d. assumptions.

Methodology: Factorized Active Querying (FAQ)

The authors propose Factorized Active Querying (FAQ), a method that combines historical data with adaptive sampling to construct tight CIs with valid coverage. FAQ consists of three core components:

1. Bayesian Factor Model for Historical Information

To leverage historical performance data (a matrix of model-question outcomes from prior models), FAQ employs a probabilistic factor model.

• Latent Factors: Each historical model $i$ is assigned a latent factor $u_i \in \mathbb{R}^k$, and each question $j$ is assigned a latent factor $v_j \in \mathbb{R}^k$.
• Likelihood: The probability of a correct answer is modeled via a logistic link: $P(H_{ij}=1) = \sigma(u_i^\top v_j)$.
• Role: This model is used only to guide adaptive query selection and provide initial predictions. It is not used to derive the final confidence intervals, ensuring that the method's validity does not depend on the factor model being perfectly specified.
• Deployment: For a new model, its latent factor $u$ is initialized from the empirical mean and covariance of historical model factors. As the model is queried, $u$ is updated online using Laplace approximations.

2. Pro-Active Inference (PAI) Estimator

To ensure valid frequentist coverage without relying on the factor model's fidelity, FAQ utilizes Pro-Active Inference (PAI), a finite-population extension of active statistical inference.

• Mechanism: Unlike sequential active inference, which processes questions in a fixed "nature's order" and decides to label or skip, PAI proactively selects the next question to query from the entire bank based on current information.
• Estimator: The estimator $\hat{\theta}_{n_b}$ is an unbiased, inverse-probability weighted estimator:
  $$\hat{\theta}_{n_b} = \frac{1}{n_b} \sum_{t=1}^{n_b} \phi_t, \quad \text{where } \phi_t = \frac{1}{N_q} \sum_{j=1}^{N_q} \hat{p}^{(t-1)}_j + \frac{1}{N_q} \frac{z_{I_t} - \hat{p}^{(t-1)}_{I_t}}{q_t(I_t)}$$
  Here, $\hat{p}^{(t-1)}_j$ is the predicted probability from the factor model, $z_{I_t}$ is the observed label, and $q_t(I_t)$ is the probability of selecting question $I_t$.
• Theoretical Guarantee: Under mild regularity conditions, the authors prove via a martingale central limit theorem (CLT) that $\hat{\theta}_{n_b}$ is unbiased and asymptotically normal, allowing for the construction of valid CIs: $[\hat{\theta}_{n_b} \pm z_{1-\alpha/2} \frac{\hat{\sigma}_{n_b}}{\sqrt{n_b}}]$.

3. Hybrid Sampling Policy

To minimize the width of the CI, FAQ employs a time-varying hybrid sampling policy $q_t(\cdot)$ that balances exploitation (variance reduction) and exploration (learning the new model's latent factor).

• Oracle Variance Reduction: Inspired by an oracle setting, questions with predicted probabilities near 0.5 (high uncertainty) are upweighted to reduce variance.
• Active Learning: Questions that are expected to maximally reduce the posterior variance of the model's latent factor $u$ are upweighted to improve future predictions.
• Mixing: The policy combines these scores with a tempering exponent and a uniform mixing term $\tau$ to ensure all questions retain a non-zero selection probability, satisfying the regularity conditions required for the CLT.

Key Contributions

1. Framework: Introduction of FAQ, a method for LLM benchmarking that achieves valid frequentist coverage under fixed budgets while leveraging historical data.
2. Pro-Active Inference: A novel extension of active inference that allows for direct, data-adaptive question selection from a finite bank, rather than sequential stream processing, while preserving statistical validity.
3. Efficiency: The method delivers up to 5× effective sample size (ESS) gains over uniform sampling and 1.8–2.4× gains over strong baselines (including post-hoc selected active inference variants). This translates to matching the CI width of uniform sampling while using up to 5× fewer queries.
4. Robustness: The method maintains valid coverage even under significant historical data missingness (e.g., only 0.1% of historical entries observed) and in cold-start scenarios where target-suite history is unavailable (using cross-suite transfer).

Experimental Results

The authors evaluated FAQ on two benchmark suites: MMLU-Pro (12K questions) and a composite suite (BBH+GPQA+IFEval+MATH+MuSR, 9.5K questions) using 2,200 test models and 100 random seeds per model.

• Fully-Observed History: FAQ achieved 4–5× higher ESS than uniform sampling and 1.8–2.4× higher ESS than the strongest post-hoc baseline. Empirical coverage remained near the 95% target.
• Sparse History: Even with only 12–42% of historical data observed, FAQ substantially outperformed baselines. With only 0.1% observed, it retained a consistent ESS edge, though performance degraded slightly at very large budgets due to noisy factor estimates.
• Cold-Start Transfer: In a setting with zero target-suite history, FAQ (initialized via MMLU-Pro transfer) achieved up to 1.32× ESS over uniform sampling, outperforming baselines that had access to 0.1% target-suite history on 8 of 10 budgets.
• Coverage Stability: Coverage was stable across model release dates and accuracy levels, indicating no systematic under-coverage for specific model types.
• Ablation: Comparing FAQ to a "traditional" sequential active inference ablation (which uses the same factor model but follows nature's order) revealed that FAQ is superior in the low-budget regime (the primary target for cost savings), while the sequential method can catch up at very large budgets.

Significance and Claims

The paper claims that FAQ provides a statistically rigorous framework for continuous monitoring of LLMs. By leveraging accumulated historical data and adding fresh outcomes, it makes uncertainty-quantified evaluation progressively more cost-effective. The authors emphasize that this is particularly valuable for organizations repeatedly comparing model variants (e.g., checkpoints, fine-tunes, quantized models) under limited query or adjudication budgets.

The significance lies in the ability to reduce evaluation costs (API credits, latency, expert time) by up to 5× without sacrificing the statistical guarantees required for high-stakes deployment decisions. The method is model-free regarding the final inference, meaning its validity does not rely on the correctness of the underlying factor model, only on the regularity of the sampling policy. The authors release their curated datasets (4.4K+ models, 21.5K+ questions) and source code to support reproducible evaluation.