Technical Summary: Correcting Prompt Dependence in LLM Benchmarks

Problem Statement

Current Large Language Model (LLM) benchmarking practices frequently misstate performance metrics and uncertainty estimates due to two critical, often unmet assumptions:

1. Sufficient Data for Classical Inference: The assumption that sample sizes are large enough to rely on Central Limit Theorem (CLT)-based inference (e.g., Wald confidence intervals) is often violated due to computational constraints, leading to unreliable sample means and p-values.
2. Prompt Independence: Standard metrics assume test prompts are independent, weighting them equally. However, empirical evidence suggests that prompts within benchmarks often exhibit semantic similarity and shared failure modes, creating dependencies. This "prompt dependence" reduces the effective number of independent samples, causing standard methods to overestimate the effective sample size and produce miscalibrated uncertainty.

Methodology: Bayesian Hierarchical Model with Embedding-Space Clustering (BHM-ESC)

To address these issues, the authors propose a corrective Bayesian Hierarchical Model with Embedding-Space Clustering (BHM-ESC). The approach operates as follows:

• Semantic Clustering: Instead of relying on predefined task labels, the model treats the number of semantic clusters ($S$) as an unknown random variable. It utilizes a prior distribution (specifically a Binomial distribution truncated at $n$) to infer $S$ directly from the data.
• Embedding Space Partitioning: Prompt embeddings are generated using a Sentence Transformer (e.g., all-MiniLM-L6-v2). Agglomerative cosine similarity clustering is applied to partition prompts into $S$ disjoint clusters based on semantic proximity.
• Hierarchical Modeling:
  • Within each cluster $k$, prompts are assumed to share a common task success probability $p_k$.
  • $p_k$ is modeled with a uniform Beta(1, 1) prior.
  • The observed successes $x_i$ for prompt $i$ (run $m$ times) follow a Binomial distribution: $x_i | p_k \sim \text{Binomial}(m, p_k)$.
  • The overall benchmark performance $\bar{p}$ is modeled as an equally-weighted mixture of the cluster-wise probabilities, effectively pooling variability at the topic level.
• Inference via Importance Sampling: Since the posterior distributions do not admit closed-form expressions, the authors employ an importance sampling scheme. They draw samples of $S$ from the prior, determine the cluster partition, sample $p_{1:S}$ from the tractable conditional posteriors, and calculate importance weights based on the marginal likelihood $\pi(x_{1:n} | S)$. This yields weighted samples approximating the posterior distributions for performance metrics and cluster counts.

Key Contributions

1. Empirical Evidence of Dependence: The paper provides quantitative evidence (via Hopkins statistics and UMAP visualizations) that popular benchmarks (including MMLU, GSM8K, HarmBench, and Garak) exhibit significant clustering structure, violating the independence assumption.
2. A Principled Correction Framework: The BHM-ESC model explicitly accounts for inter-prompt dependence without requiring predefined cluster labels or task metadata. It infers cluster structure directly from embedding spaces.
3. Robustness in Limited Data: The approach is designed for limited-data regimes where CLT-based methods fail, providing calibrated posterior uncertainty estimates.
4. Internal Validity: The inferred cluster structures align with human judgments of semantic similarity and independence, as verified through Posterior Similarity Matrices (PSMs) and comparison with "oracle" baselines using ground-truth labels.

Results

The authors evaluated BHM-ESC on adversarial robustness benchmarks (Garak subsets: ANSI Escape, Repeat, Latent Injection, JavaScript) using two LLM architectures (Pythia-2.8B and Mamba-2.8B).

• Performance Improvements: Compared to baselines (naive frequentist, Bayesian models without clustering, and Bayesian models with fixed $S=1$ or $S=n$), BHM-ESC achieved:
  • 4–73% reduction in Mean Absolute Errors (MAE) for posterior predictive means.
  • 20–450 unit improvements in Expected Log Posterior Density (ELPD), indicating superior model fit and uncertainty quantification.
• Effective Sample Size Correction: The model demonstrated that baseline approaches systematically overestimate the number of independent samples by factors of 1.3 to 5.6 times. BHM-ESC correctly identified the reduced effective sample size ($S$) relative to the nominal prompt count ($n$).
• Impact on Conclusions: The corrected metrics materially altered performance comparisons. For instance, BHM-ESC suggested that robustness to ANSI attacks was more homogeneous and certain than indicated by naive baselines.
• Embedding Agnosticism: Performance improvements were consistent across different embedding methods (Sentence Transformers vs. TF-IDF), suggesting the gains stem from structural recovery rather than specific embedding artifacts.

Significance and Claims

The paper claims that prompt dependence is a systematic issue in LLM benchmarking that introduces bias into summary statistics and uncertainty estimates. By treating the number of clusters as unknown and learning structure directly from embeddings, BHM-ESC offers a benchmark-agnostic solution that:

• Recovers clustering structures consistent with human intuition.
• Provides calibrated uncertainty estimates in data-scarce settings where traditional statistical methods are invalid.
• Prevents the overconfidence in statistical conclusions that arises from treating semantically similar prompts as independent data points.

The authors note that while the approach improves estimation accuracy, it does not require exhaustive or balanced task coverage to produce calibrated estimates, as the dependence is inferred and corrected for automatically. Future work could explore relaxing the assumption of identical success probabilities within clusters or extending the method to multi-turn benchmarks.