📊 statistics

BayesAME: Bayesian Active Model Evaluation

BayesAME is a sequential Bayesian framework that automatically determines an optimal coreset size for efficiently evaluating large generative models by modeling latent item abilities, quantifying uncertainty, and iteratively selecting informative items until performance estimates stabilize, while demonstrating superiority over random selection and existing methods through the use of continuous log-likelihoods.

Original authors: Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet, Virginia Aglietti, Silvia Chiappa

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet, Virginia Aglietti, Silvia Chiappa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge how good a new video game character is. You could play through the entire 100-hour campaign, fighting every boss and solving every puzzle, but that takes forever and burns out your battery. Instead, you decide to play just a few levels to get a feel for the character's strength. This is the daily struggle for scientists testing massive Artificial Intelligence models: running them on every single question in a giant test bank is too slow and expensive. So, they try to pick a tiny "sample" of questions to guess the final score.

The tricky part is which questions to pick. If you pick them randomly, like grabbing a handful of M&Ms from a jar, you might get lucky, or you might get a handful of only red ones and miss the blue ones entirely. For a long time, many experts thought that because AI models are so complex, picking questions randomly was just as good as trying to be clever about it. They believed that the only way to be sure was to just pick a random handful and hope for the best. But what if there was a way to be a smart detective, looking at how other similar characters performed in the past to figure out exactly which few questions would tell you the most about the new one?

This is the story of BayesAME, a new method developed by researchers at Google DeepMind and Imperial College London. Think of BayesAME as a super-smart, curious librarian who wants to know how good a new author is without reading their entire book. Instead of reading page by page, the librarian looks at the author's previous work (or similar authors' work) to guess which specific paragraphs will reveal the most about their writing style.

Here is how the magic works: The researchers treat the AI's performance not as a fixed number, but as a mystery box with a "latent ability"—a hidden skill level that changes depending on the type of question. They use a mathematical tool called a "Gaussian prior" (a fancy way of saying "a smart guess based on history") to believe that if the new AI acts like old, known AIs on certain questions, it will likely act similarly on new ones. As the librarian reads a few pages (evaluates a few questions), they update their guess. If the new AI surprises them, they adjust their map.

The coolest part is that BayesAME doesn't need you to say, "Read exactly 50 pages." Instead, it has its own internal compass. It keeps reading one page at a time, asking, "If I read this next page, will it clear up my confusion?" It stops only when it is so confident in its estimate that reading more pages wouldn't change its mind much. It's like a detective who stops investigating once the clues are so clear that the suspect's guilt is obvious, rather than sticking to a rule of "investigate for exactly 3 hours."

The paper finds that this smart, detective-style approach is significantly better than the old "grab a handful randomly" method. In their tests across many different benchmarks (like GPQA, MMLU-Pro, and others), BayesAME consistently guessed the final score with much higher accuracy using fewer questions. It proved that being clever about which questions to ask really does matter, overturning the idea that random guessing is just as good.

Furthermore, the researchers discovered that the type of score matters. If you just ask "Right or Wrong?" (binary scores), it's harder to be precise. But if you use the model's actual confidence levels (continuous scores, like "I'm 87% sure this is right"), BayesAME becomes even sharper, like switching from a black-and-white sketch to a high-definition photo. They also showed that if you are testing multiple AI models at once, you can save even more time by noticing if they tend to make the same mistakes together, allowing you to learn about several models with a single set of questions.

In short, the paper suggests that we don't need to waste time testing AI on everything. By using a smart, step-by-step strategy that learns from history and stops exactly when it has enough proof, we can get reliable answers faster and cheaper. It's a win for efficiency, proving that sometimes, the best way to know the whole story is to ask the right questions, not just ask more questions.

Technical Summary: BayesAME: Bayesian Active Model Evaluation

Problem Statement

Evaluating large generative models across comprehensive benchmarks is computationally expensive and time-consuming. Current approaches often mitigate this by evaluating models on a subset of items, known as a coreset, to estimate full benchmark performance. However, existing literature predominantly requires practitioners to manually specify the coreset size. This creates a dilemma: when reliable performance estimation is prioritized over strict efficiency, there is no mechanism to automatically determine the necessary coreset size. Furthermore, recent literature (e.g., Zhang et al., 2025) has expressed skepticism regarding the utility of non-random coreset selection, particularly in extrapolation regimes where target models significantly deviate from the performance range of historical reference models, suggesting that random sampling often suffices.

Methodology: BayesAME

The authors propose BayesAME, a sequential Bayesian framework designed to automatically determine the optimal coreset size while leveraging historical reference models.

Core Framework

Latent Ability Modeling: The method groups benchmark items into "buckets" based on identical performance vectors from $K_r$ reference models. Each bucket $b$ is assigned a latent ability $\theta_b$ representing the target model's underlying performance on that group.
Prior Distribution: A joint Gaussian prior is placed over these abilities. The prior mean is the average of the reference scores for the bucket, and the covariance encodes the belief that buckets with similar reference scores yield correlated latent abilities for the target model.
Sequential Process:
- Estimation: The posterior distribution of latent abilities is updated as items are evaluated. This yields a performance estimator $\hat{R}$ and a quantification of uncertainty (variance).
- Active Selection: In regimes where reference scores are sufficiently unique (nearly unique reference regime), the method selects the next item to evaluate based on expected information gain, specifically maximizing the reduction in the variance of the full benchmark performance estimate.
- Fallback: In regimes with non-unique reference scores (e.g., binary scores with small reference pools), the method defaults to random selection to avoid bias, ensuring the expected number of sampled items remains proportional to bucket sizes.
Stopping Criterion: The process iteratively augments the coreset until two conditions are met:
- The performance estimate $\hat{R}$ stabilizes (fluctuation over a window $W$ falls below a threshold $\epsilon_1$ ).
- The uncertainty (width of the 95% credible interval) falls below a user-defined threshold $\epsilon_2$ .

Multi-Target Extension

For scenarios involving multiple target models, BayesAME employs a linear model of coregionalization. This assumes that the latent abilities of different target models are driven by shared, independent latent random vectors. This allows the framework to capture correlations between models, enabling the simultaneous evaluation of multiple targets with a reduced total coreset size compared to evaluating them independently.

Computational Efficiency

To ensure scalability, the method utilizes a bucket-level formulation. Instead of inverting an $n \times n$ matrix (where $n$ is the coreset size), it inverts at most a $B \times B$ matrix (where $B$ is the number of unique reference score vectors). The computational complexity per iteration is $O(\min(n, B)^2)$ .

Key Contributions

Automatic Coreset Determination: Formulates efficient model evaluation as a Bayesian sequential process that automatically determines the coreset size based on user-defined precision and stability thresholds, rather than requiring a fixed input.
Robust Estimators: Develops computationally efficient performance estimators with well-defined properties, including a control-variate estimator for non-unique reference regimes.
Active Selection Strategies: Designs selection strategies that identify benchmark items maximizing information gain, dynamically switching between active and random selection based on the uniqueness of reference signals.
Multi-Target Correlation: Extends the framework to capture performance correlations across multiple target models, further reducing evaluation costs.
Empirical Validation: Conducts extensive experiments across diverse benchmarks (GPQA, MMLU-Pro, BBH, etc.), demonstrating that BayesAME consistently outperforms sequential adaptations of existing methods (e.g., AIPW, ProEval, IRT) in both interpolation and extrapolation regimes.
Refutation of Skepticism: Provides comprehensive analysis overturning recent claims that non-random selection offers no advantage. The paper demonstrates that when leveraging richer information (continuous scores, larger reference pools), non-random selection significantly outperforms random sampling and the random sample mean baseline.
Score Utilization: Highlights that leveraging continuous response log-likelihoods significantly enhances estimation accuracy compared to traditional binary scores.

Results

Performance: Across seven benchmarks and varying proportions of reference models (10%, 50%, 90%), BayesAME consistently achieves the lowest Root Mean Square Error (RMSE) and the highest RMSE Gain relative to the random sample mean baseline.
Cost-Accuracy Trade-off: BayesAME establishes a strictly superior Pareto frontier, achieving the same level of estimation accuracy at a significantly lower coreset size compared to baselines.
Multi-Target Efficiency: In high-correlation regimes, the multi-target extension achieves steeper declines in error and reaches high accuracy with fewer evaluations than the single-target approach. In low-correlation regimes, the method degrades gracefully, matching the performance of competitive baselines without catastrophic failure.
Robustness: The method remains advantageous even when the coreset size is predetermined, and it effectively handles both interpolation (target models within reference range) and extrapolation (target models outside reference range) scenarios.

Significance and Claims

The paper claims to address a critical gap in model evaluation by providing a method that prioritizes reliable performance estimation over fixed computational budgets. By automating the determination of the coreset size, BayesAME allows practitioners to define their required precision (via $\epsilon_1$ and $\epsilon_2$ ) rather than guessing the necessary sample size.

Crucially, the authors position their work as a direct rebuttal to recent literature suggesting that sophisticated active selection is unnecessary. They demonstrate that:

Non-random selection is advantageous: When sufficient reference data and continuous scores are available, active selection targets informative items effectively, outperforming random baselines.
Richer signals matter: The use of continuous log-likelihoods over binary scores is a key factor in enabling effective active selection and reducing estimation error.
Robustness in Extrapolation: Contrary to claims that extrapolation regimes render active selection ineffective, BayesAME maintains superior performance even when target models deviate significantly from reference models, provided the reference pool is sufficiently diverse.

The authors conclude that while simple methods are difficult to outperform, their Bayesian framework successfully leverages reference model information and continuous scores to provide a more efficient and reliable evaluation process, particularly when the goal is high-confidence performance estimation.