💬 NLP

Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation

This paper proposes a principled Bayesian framework that replaces the unstable Pass@k metric with posterior estimates and credible intervals to provide more reliable, compute-efficient, and uncertainty-aware rankings for large language models across both binary and graded evaluation scenarios.

Original authors: Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, Vipin Chaudhary

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohsen Hariri, Amirhossein Samandar, Michael Hinczewski, Vipin Chaudhary

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a talent scout trying to find the best math student in a room full of 20 geniuses. You give them a hard problem.

The Old Way (Pass@k): The "Lucky Guess" Lottery
Currently, most people evaluate AI models using a method called Pass@k. Imagine you ask a student to solve a math problem. They might get it right on the first try, or they might get it wrong five times and then get it right on the sixth.

Pass@k says: "If the student gets it right at least once in their first $k$ attempts, they pass."
The Problem: This is like judging a basketball player based on whether they made one lucky shot out of ten tries. If you only let them take 5 shots, the results are chaotic. One day they might hit a lucky shot and look like a pro; the next day they might miss every shot and look terrible, even if their actual skill hasn't changed. It's noisy, unstable, and often misleading, especially when you don't have the budget to let them take 100 shots.

The New Way (Bayes@N): The "Confidence Coach"
This paper proposes a new framework called Bayes@N. Instead of just counting "Pass" or "Fail," it acts like a wise coach who keeps a running diary of the student's performance.

Here is how it works, using simple metaphors:

1. The "Categorical" Scorecard (Not Just Right/Wrong)

In the old days, a test was just a 1 (Right) or 0 (Wrong).

The New Way: Imagine a rubric that says:
- 0: The student refused to answer.
- 1: They tried but got the math wrong.
- 2: They got the math right but wrote it in the wrong format (like forgetting to box the answer).
- 3: They got it right and formatted it perfectly.
- 4: They got it right, formatted it, and explained their steps clearly.
  The new system treats these as different "flavors" of performance, not just a binary pass/fail. It gives you a much richer picture of how the model thinks.

2. The "Confidence Interval" (The Safety Net)

This is the most important part.

The Old Way: If Model A scores 60% and Model B scores 61%, the old system says, "Model B is the winner!" But was that 1% difference real, or just luck? The old system doesn't know.
The New Way: The Bayesian framework draws a "fuzzy circle" (called a Credible Interval) around the score.
- If Model A's circle is [55% to 65%] and Model B's circle is [56% to 66%], the circles overlap. The coach says, "Stop! We can't tell who is better yet. It's a tie."
- If the circles don't touch, then we declare a winner with confidence.
  This prevents the "leaderboard churn" where models jump up and down in rank every time you run the test one more time.

3. The "Prior Knowledge" (Learning from the Past)

Imagine you are evaluating a new version of a student who just took a test last year.

The Old Way: You ignore last year's results and start from scratch.
The New Way: You use last year's performance as a "head start" (a Prior). If the student was great last year, you start with a slight assumption they are good this year, and you only need a few new test results to confirm it. This saves a massive amount of time and money.

4. Why This Matters (The "Gold Standard")

The authors ran simulations where they knew the true skill level of the models (like knowing the exact weight of a coin). They found that:

Pass@k (the old way) was like trying to guess the weight of a coin by flipping it 5 times. It took a huge number of flips to get a stable answer.
Bayes@N (the new way) was like using a scale. It found the true ranking much faster and with far fewer flips.

The Big Takeaway

The paper argues that we should stop treating AI evaluation like a lottery ticket (did it get lucky this time?) and start treating it like a statistical investigation.

By using this new Bayesian framework, we can:

Save Money: We need fewer computer runs (trials) to know who is actually the best.
Be Honest: We can admit when two models are too close to call, rather than forcing a fake winner.
Be Nuanced: We can reward models for being "mostly right" or "well-formatted," not just for getting a lucky perfect answer.

In short: Don't just count the wins; measure the confidence. This makes AI evaluation fairer, cheaper, and more reliable.

1. Problem Statement

Current Large Language Model (LLM) evaluation, particularly for reasoning tasks (e.g., math, code), relies heavily on the Pass@k metric and simple average accuracy (avg@N). The authors identify several critical flaws in these approaches:

Instability: Rankings are highly sensitive to the number of trials ( $N$ ) and decoding seeds, leading to unstable leaderboards, especially on small benchmarks (e.g., AIME with ~30 problems).
Lack of Uncertainty Quantification: Pass@k provides a point estimate without principled confidence intervals. Determining if a performance gap is statistically significant or mere noise is difficult.
Binary Limitation: Standard metrics reduce evaluation to binary (correct/incorrect), discarding valuable information about partial correctness, formatting errors, or reasoning quality.
Computational Inefficiency: Achieving stable rankings often requires an impractically large number of trials ( $N$ ), making evaluation expensive.

2. Methodology: The Bayesian Framework

The paper proposes a principled Bayesian evaluation framework that treats LLM performance as a statistical inference problem.

Core Concepts

Categorical Modeling: Instead of binary outcomes, the framework models responses as categorical variables with $C+1$ categories (e.g., correct, partially correct, formatting error, refusal).
Dirichlet-Multinomial Inference:
- Prior: A Dirichlet prior is placed over the probability vector $\pi_\alpha$ for each question $\alpha$ . A uniform prior (no prior knowledge) is the default, but informative priors can be incorporated.
- Posterior: Given observed trial results, the posterior distribution is derived analytically.
- Estimators: The framework provides closed-form expressions for the posterior mean ( $\mu$ ) and variance ( $\sigma^2$ ) of any weighted performance metric.
Algorithm (Bayes@N):
- Inputs: A results matrix $R$ ( $M$ questions $\times$ $N$ trials) and a weight vector $w$ .
- Output: A point estimate ( $\mu$ ) and a standard deviation ( $\sigma$ ) representing uncertainty.
- Complexity: Linear in the number of trials and categories ( $O(M(N+C))$ ), requiring no Monte Carlo sampling for the core estimates.

Key Mechanisms

Credible Intervals (CI): The framework calculates exact credible intervals for performance scores. A "winner" is only declared if the CIs of two models do not overlap, providing a transparent decision rule.
Ranking with Uncertainty: Models are ranked by their posterior means, but ties are assigned to models whose CIs overlap. This prevents over-interpretation of small, noisy gaps.
Convergence Analysis: The paper defines convergence@n as the minimum number of trials required for a ranking to match a "gold standard" (derived from a large $N_{max}$ ).
Prior Incorporation: The framework allows reusing data from related tasks or previous model versions as a prior ( $R_0$ ), potentially accelerating convergence for new evaluations.

3. Key Contributions

Unified Framework: Replaces Pass@k and avg@N with a single Bayesian estimator that handles both binary and graded (rubric-based) evaluations seamlessly.
Analytical Uncertainty: Provides exact, closed-form credible intervals without relying on the Central Limit Theorem (CLT), which is often invalid for small sample sizes common in LLM benchmarks.
Compute-Efficient Protocol: Demonstrates that reliable rankings can be achieved with significantly fewer trials than Pass@k methods, reducing computational costs.
Scorio Library: Releases an open-source Python package (scorio) implementing the framework, including support for custom rubrics and priors.

4. Experimental Results

The authors validated the framework using simulations with known ground truths and real-world math reasoning benchmarks (AIME'24/25, HMMT'25, BrUMO'25).

Faster Convergence:
- Bayes@N converges to the true ranking (measured by Kendall's $\tau$ ) much faster than Pass@k variants.
- On HMMT'25 and BrUMO'25, Bayes@N achieved mean convergence at ~44 and ~27 trials, respectively, whereas Pass@k required ~70 and ~48 trials.
- On AIME'25, Pass@k methods failed to converge within 80 trials, while Bayes@N provided stable rankings with quantified uncertainty.
Rank Stability:
- In simulations with 11 models, Bayes@N reached high correlation with the gold standard ( $\tau > 0.9$ ) by $N=10$ , while Pass@k variants lagged significantly.
- As the number of models increased (up to 20), Pass@k rankings became increasingly unstable and often failed to converge, whereas Bayes@N maintained robustness.
Handling Ties and Noise:
- The framework correctly identified when performance gaps were statistically indistinguishable (overlapping CIs), preventing false claims of superiority.
- In the "worst-case" bootstrap scenarios, Bayes@N provided a mechanism to quantify uncertainty even when rankings did not fully stabilize.
Rubric-Aware Evaluation:
- The framework successfully applied to 12 different categorical schemas (e.g., penalizing "confidently wrong" answers, rewarding formatting).
- Results showed that model rankings are sensitive to the chosen rubric, highlighting the need for explicit, weighted evaluation rather than simple correctness.

5. Significance and Implications

Paradigm Shift: The paper argues for moving away from "Pass@k" as the standard metric toward a posterior-based, uncertainty-aware protocol.
Resource Efficiency: By enabling reliable comparisons with fewer trials, the framework reduces the computational cost of LLM evaluation, making it more accessible and sustainable.
Scientific Rigor: It introduces a transparent decision rule for declaring model superiority (non-overlapping CIs), reducing leaderboard churn and the "noise" of minor performance fluctuations.
Flexibility: The ability to incorporate prior knowledge and handle complex, graded rubrics makes the framework applicable to a wide range of evaluation scenarios beyond simple correctness, including safety, alignment, and reasoning quality.

In conclusion, the authors demonstrate that Bayes@N offers a statistically rigorous, computationally efficient, and more stable alternative to existing LLM evaluation metrics, fundamentally improving how the community measures and compares model capabilities.