Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation

Technical Summary: Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation

Problem Statement
Current Large Language Model (LLM) benchmark evaluations predominantly rely on a single generation per prompt, utilizing either deterministic greedy decoding or a single random sample. The authors argue that this paradigm overlooks the inherent stochasticity of LLMs. Deterministic generation misaligns with real-world applications where randomness is intrinsic, potentially leading to biased performance estimates. Conversely, relying on a single random sample introduces significant sampling variance, resulting in unreliable benchmark scores, particularly when dataset sizes are small. Furthermore, single-generation approaches fail to provide fine-grained insights into individual prompt difficulty or the semantic consistency of model responses, limiting the ability to diagnose dataset quality or model weaknesses.

Methodology
The paper proposes a hierarchical statistical model to reframe benchmark evaluation as an estimation problem that explicitly incorporates both benchmark characteristics and LLM randomness.

1. Hierarchical Model Formulation:

   • The benchmark is modeled with $n$ prompts. For each prompt $i$, a latent difficulty parameter $p_i$ (the probability of a correct answer) is drawn from an unknown distribution $P(\mu, \sigma; \theta)$, where $\mu$ is the mean benchmark score and $\sigma$ represents the variability in prompt difficulty.
   • For each prompt, $k$ generations are produced. The correctness of the $j$-th generation for prompt $i$, denoted $y_{i,j}$, follows a Bernoulli distribution with parameter $p_i$.
   • The authors derive moment estimators for the prompt-level probability $\hat{p}_i$ and the overall benchmark score $\hat{\mu}$.

2. Variance Analysis:

   • Using Lemma 2.1, the paper theoretically demonstrates that the variance of the benchmark score estimator $\hat{\mu}$ can be decomposed into "within-prompt variance" and "between-prompt variance."
   • Crucially, the within-prompt variance decreases as the number of generations $k$ increases, converging to zero as $k \to \infty$. In contrast, the between-prompt variance remains unaffected by $k$. This proves that increasing $k$ reduces the overall estimation error.

3. Prompt-Level Metrics:

   • $P(\text{correct})$: Defined as the estimated probability $\hat{p}_i = \frac{1}{k}\sum_{j=1}^k y_{i,j}$. This serves as a fine-grained difficulty score for individual prompts.
   • Semantic Consistency ($S(\text{consistency})$): To detect labeling errors, the authors introduce a metric based on the entropy of semantic clusters. Responses are clustered into $C$ semantic sets, and consistency is measured as $S(\text{consistency}) = \sum_{c=1}^C \text{Prop}_c \log \text{Prop}_c$. High consistency (low entropy) combined with low $P(\text{correct})$ suggests a prompt may be mislabeled or ambiguous.

4. Data Mapping:

   • The authors visualize the relationship between $P(\text{correct})$ and $S(\text{consistency})$ to create a "data map." This map aids in identifying outliers, such as prompts where the model generates consistent but incorrect answers, indicating potential dataset noise.

Key Results
The methodology was evaluated across four benchmarks (MMLU-Pro, GSM8K, MuSR, IFEval) using four open-source LLMs (Llama 3.1 8B/70B, Qwen 2.5 7B, Ministral 8B) with $k=50$ generations.

• Variance Reduction: Increasing $k$ significantly narrows the 95% confidence intervals of benchmark scores. For example, on GSM8K, the performance gap ($\Delta$) between the best and worst single-generation runs was as high as 18.6 for Llama 3.1 8B, whereas the $k=50$ estimate provided a stable score with a standard error of 0.68.
• Greedy vs. Random Sampling: The study found that greedy decoding and random sampling often yield different benchmark scores. For instance, on MuSR, Llama 3.1 8B scored 24.8 with greedy decoding but 29.0 with random sampling ($k=50$). The authors note that even a modest number of sampled generations often approximates the true $P(\text{correct})$ better than greedy decoding.
• Model Behavior Analysis:
  • Models exhibited "diffuse density" distributions of $P(\text{correct})$ on reasoning-heavy tasks (MMLU-Pro, IFEval, MuSR), behaving similarly to random samplers.
  • Simpler tasks (GSM8K) showed more stable distributions with tails near 0 and 1.
  • Larger models (Llama 70B) demonstrated more stable reasoning processes across all benchmarks compared to smaller models.
• Error Detection: In a case study on GSM8K, the data map identified 18 prompts with low $P(\text{correct})$ and high $S(\text{consistency})$. Manual review revealed that 44.4% of these were mislabeled or ambiguous, validating the utility of multiple generations for dataset cleaning.

Significance and Claims
The paper claims that leveraging multiple generations transforms benchmark evaluation from a point estimate into a robust statistical process. The primary contributions are:

1. Theoretical Validation: Providing a mathematical proof that multiple generations reduce estimation variance, addressing the unreliability of single-sample benchmarks.
2. Granular Diagnostics: Introducing $P(\text{correct})$ as a prompt-level difficulty metric, enabling a deeper understanding of model strengths and weaknesses beyond aggregate scores.
3. Quality Control: Demonstrating that the combination of multiple generations and semantic consistency metrics can effectively detect mislabeled or ambiguous prompts, offering a scalable tool for improving benchmark construction.

The authors maintain a modest tone regarding limitations, acknowledging that multiple generations increase computational costs and that their current error detection method has a true positive rate of approximately 50%. They suggest future work should explore minimal generation requirements for robust evaluation and more sophisticated semantic metrics for error detection.