Technical Summary: Active Testing of Large Language Models via Approximate Neyman Allocation

Problem Statement

The evaluation of Large Language Models (LLMs) and multimodal models has become a recurring, high-cost bottleneck. As models scale through pre-training, post-training, and test-time paradigms, the computational cost per inference rises. Simultaneously, the complexity of target tasks often exceeds the capabilities of standard annotators, necessitating costly domain experts for labeling.

Existing active testing approaches, which aim to estimate full-pool performance from a small, informative subset of data, are primarily designed for classification tasks. These methods typically rely on:

1. Classification-style entropy: Using token-level predictive entropy (PE) as an uncertainty signal.
2. LURE Estimators: Importance-weighted estimators that require retraining data selectors or access to internal model activations/logits (often unavailable in black-box API settings).

The authors identify that these existing pipelines fail on modern generative tasks because:

• Semantic vs. Surface Uncertainty: Token-level PE conflates uncertainty about meaning with uncertainty about surface form (paraphrasing). Semantically equivalent generations with different phrasings inflate entropy without reflecting genuine difficulty.
• Overconfidence: Modern LLMs are systematically overconfident, collapsing token-level entropy into a narrow low-entropy band, weakening the acquisition signal.
• Estimator Instability: The LURE estimator's importance weighting mechanism amplifies mismatches between surrogate and target models, leading to variance rather than reduction.

Methodology

The paper proposes a novel, training-free, and theoretically unbiased active testing framework tailored for generative and multimodal tasks. The method, detailed in Algorithm 1, operates in four steps:

1. Signal Generation via Surrogate Models

Instead of relying on the target model's internal states, the method uses a smaller, cheaper surrogate model ($g$) to generate $k$ responses for each input in the evaluation pool $D$.

• Semantic Entropy (SE): To address the token-level vs. meaning-level issue, the method clusters the surrogate's generations into meaning-equivalence classes. It computes the Shannon entropy over the probabilities of these classes. This captures uncertainty regarding the answer rather than the phrasing.
• Self-Consistency Score (SC): A secondary signal measuring the agreement among the surrogate's generations. High SC indicates consensus (easier instances), while low SC indicates disagreement.

2. Stratification

The evaluation pool $D$ is partitioned into $H$ strata based on the Semantic Entropy of the surrogate model.

• A base stratum ($D_0$) is created for instances where $SE = 0$ (highly certain).
• Remaining instances ($SE > 0$) are binned using equal-quantile binning.
• This creates strata where the variance of the target loss is expected to be relatively homogeneous.

3. Approximate Neyman Allocation

Classical Neyman allocation minimizes estimation variance by allocating labels proportional to the stratum size ($N_h$) and the stratum standard deviation ($\sigma_h$). However, $\sigma_h$ is unknown without labeling the entire pool.

• Proxy Strategy: The method approximates $\sigma_h$ using the mean self-consistency score ($p_h$) of the surrogate within each stratum.
• Allocation Formula: The label budget $m_h$ for stratum $h$ is allocated as:
  $$m_h \propto N_h \cdot \sqrt{p_h(1 - p_h) + \delta}$$
  where $\delta$ is a smoothing offset to prevent collapse when $p_h \to 1$. This allows the method to allocate more budget to strata where the surrogate indicates high uncertainty (low self-consistency), without requiring target labels.

4. Estimation

The method samples $m_h$ items uniformly from each stratum and labels them using the target model. It aggregates the results using a Horvitz–Thompson estimator, which is proven to be theoretically unbiased for the finite-pool risk.

Key Contributions

1. First Unbiased Generative Active Testing Pipeline: To the authors' knowledge, this is the first method providing a theoretically unbiased, label-efficient estimator specifically for generative language and multimodal evaluation that supports training-free operation and API-only deployment.
2. Surrogate-Driven Semantic Entropy: The introduction of a new algorithm that leverages semantic entropy from surrogate models to structure the sampling process, effectively decoupling semantic uncertainty from surface-form variation.
3. Proxy-Neyman Allocation: A novel allocation strategy that couples semantic entropy signals with a variance-aware budget distribution, eliminating the need for retraining selectors or accessing internal model logits.

Empirical Results

The method was evaluated across four benchmarks (MMLU-Redux, MMLU-Pro, GPQA-Diamond, MMBench) using various open-source (Qwen-3.5 series) and proprietary (GPT-5.4 series) model pairs.

• Performance: The method consistently outperforms Uniform Sampling and Proportional Allocation. In the representative setting ($M=70$), it achieved up to a 28% reduction in Mean Squared Error (MSE) compared to Uniform Sampling.
• Oracle Proximity: The method closely tracks the performance of Oracle-Neyman (the theoretical optimum that requires full knowledge of target variances), demonstrating that the surrogate-driven proxy effectively captures variance information.
• Budget Savings: At matched precision, the method achieved an average of 22.9% savings in expert labeling costs across benchmarks. Savings ranged from 15.0% (MMBench) to 28.0% (MMLU-Redux).
• Ablation Studies:
  • Stratification: Adaptive SE stratification (isolating zero-SE clusters) outperformed Equal-width, k-means, and standard Quantile binning.
  • Allocation: The Proxy-Neyman strategy significantly outperformed Equal, Proportional, and Power allocation rules, confirming that variance-aware allocation is critical.

Significance and Claims

The paper claims that active testing for generative models requires a fundamental shift from token-level metrics to meaning-level metrics. By moving from classification-style entropy to semantic entropy and replacing unstable importance-weighting estimators with stratified Horvitz–Thompson estimation, the authors demonstrate that it is possible to achieve near-optimal evaluation efficiency without retraining or access to black-box internals.

The work addresses a critical practical bottleneck: the rising cost of expert annotation for complex reasoning tasks. By reducing the number of required expert labels by ~23% on average while maintaining evaluation precision, the method offers a scalable solution for the continuous evaluation of large-scale generative models.

Limitations Acknowledged:

• The method currently relies on benchmarks with verifiable ground-truth answers (closed or parseable answer spaces). Extending this to open-ended generation remains challenging due to evaluation noise.
• The approach assumes access to a surrogate model that is cheaper than the target but correlated with target difficulty, which may limit deployment in scenarios where no suitable surrogate exists.