📊 statistics

Conditional Evaluation of Language Models with Cheap Auxiliary Signals

This paper introduces LACE, a semi-supervised estimator that leverages cheap, potentially biased auxiliary signals to efficiently and unbiasedly estimate conditional language model performance profiles without requiring gold labels for every item.

Original authors: Zhi Zhang, Lingfeng Lyu, Yue Kang, Doudou Zhou

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhi Zhang, Lingfeng Lyu, Yue Kang, Doudou Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, measuring how well a computer program understands the world has become a complex challenge. Scientists do not just want to know if a model is generally smart; they need to understand exactly where it shines and where it falters. Does a language model excel at solving algebra problems but stumble over geometry? Is it reliable for high school science questions but brittle when faced with elementary grade material? To answer these questions, researchers break down a model's performance into specific profiles, such as difficulty levels or subject categories. However, determining the true accuracy for each of these specific groups is expensive and slow. It often requires human experts to grade every single answer, a process that is too costly to do for the thousands of items in modern benchmarks.

Fortunately, there are cheaper ways to gather information about these answers. Modern systems can use other artificial intelligence models to judge responses, compare answers against each other, or ask the model how confident it feels about its own work. These signals are fast and cheap to collect for every single item, but they are imperfect. They can be biased, easily confused by the way a question is phrased, or simply wrong about certain types of problems. The central question for researchers has been whether these flawed, inexpensive signals can still help improve our understanding of a model's true performance without needing to replace the expensive human grading entirely.

A team of statisticians and computer scientists has developed a new method called LACE, which stands for Local Augmented Control-Variate Evaluation, to solve this problem. Instead of trying to force the cheap signals to be perfect, the researchers designed a system that uses them to reduce noise while keeping the expensive human grades as the anchor of truth. The core idea is to look at specific groups of questions, such as all the difficult math problems, and see how the cheap signals behave within that specific group. By comparing the average cheap signal score for the entire pool of questions against the average score for only the questions that were actually graded by humans, the method can estimate the true accuracy more precisely. It works by subtracting the predictable part of the cheap signal from the human grades, effectively removing the "static" and leaving a clearer picture of the model's actual skill.

The researchers tested this approach on eight different benchmarks covering math, science, and general knowledge, using three different large language models. They simulated a scenario where only a small fraction of the answers—ranging from 50 to 200 items out of 500—were graded by humans, while the cheap signals were available for all items. The results were striking. The new method improved the precision of the performance estimates by a factor of roughly five to six times compared to using human grades alone. In some specific cases, the improvement was even higher, reaching up to eleven times more efficient. This means that to get the same level of confidence in the results, researchers would need far fewer expensive human labels, or they could get much more detailed insights for the same cost.

Crucially, the study showed that this improvement holds even when the cheap signals are biased or miscalibrated. The method does not require the AI judges to be perfectly accurate; it only requires that they explain some of the variation in the human grades within specific groups. The researchers also demonstrated that the method works for comparing two different models directly and for adjusting scores to match how a model might perform in a real-world setting, not just on a test. They proved mathematically that the method is unbiased and that it adapts automatically to the quality of the cheap signals available. If the cheap signals are very helpful in a certain area, the method leans heavily on them; if they are useless in another area, the method ignores them and relies on the human grades.

The findings suggest a practical path forward for evaluating artificial intelligence. By combining a small amount of high-quality human feedback with a large amount of noisy, cheap data, researchers can build a much more detailed map of a model's capabilities. This allows for finer-grained decisions about which models to use for specific tasks, such as medical diagnosis or educational tools, without the prohibitive cost of grading every single interaction. The work confirms that expensive gold-standard labels do not need to be the only source of truth; when used wisely alongside abundant auxiliary data, they can reveal a much clearer and more efficient picture of how intelligent systems truly perform.

Technical Summary: Conditional Evaluation of Language Models with Cheap Auxiliary Signals

1. Problem Statement

Large Language Model (LLM) evaluation is increasingly shifting from aggregate accuracy scores to conditional performance profiles, which assess model capability across specific subgroups defined by metadata (e.g., difficulty levels, subject categories, or task types). However, estimating these profiles $\theta_m(z) = \Pr(Y_{im}=1 \mid Z_i=z)$ is challenging because gold labels ( $Y_{im}$ ) are expensive to obtain, often requiring expert grading or repeated adjudication. Consequently, estimates based on limited labeled subsets are often too noisy for fine-grained decision-making.

Conversely, modern evaluation pipelines generate abundant cheap auxiliary signals ( $S_{im}$ ) for every benchmark item, such as LLM-as-a-judge scores, pairwise comparisons, self-reported confidence, and judge disagreement. While these signals are scalable, they are frequently biased, miscalibrated, or prompt-sensitive. Treating them as calibrated probabilities is dangerous. The core statistical challenge is to leverage these uncalibrated, cheap signals to reduce the variance of conditional performance estimates without requiring them to be unbiased or calibrated, provided they explain local variation in gold correctness.

2. Methodology: LACE

The paper proposes LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation. The method operates under a setup where a gold outcome $Y$ is observed only on a small labeled subset $L \subset T$ , while a vector of cheap signals $S$ and a profiling covariate $Z$ are observed for the entire pool $T$ .

Core Principles:

Local Centering: The method relies on the identity that for any deterministic vector $b(z)$ , the conditional expectation $E[Y_{im} - b(z)^\top \{S_{im} - \mu_{S,m}(z)\} \mid Z_i = z] = \theta_m(z)$ . By centering the cheap signals $S$ by their conditional mean $\mu_{S,m}(z)$ within the target profile region, the augmentation term has a zero conditional mean. This ensures that the coefficient $b(z)$ controls efficiency (variance reduction) but not identification (bias).
Local Ridge Control Variate: LACE constructs an estimator that combines a labeled residual mean with a full-pool cheap-signal mean:
$\hat{\theta}_m^{LACE}(z) = \bar{Y}_{m,L}(z) - \hat{\beta}_m(z)^\top \{ \bar{S}_{m,L}(z) - \bar{S}_{m,T}(z) \}$
Here, $\bar{Y}_{m,L}(z)$ is the local weighted mean of gold labels, and the term in braces represents the difference between the local mean of cheap signals in the labeled set and the full pool.
Coefficient Estimation: The coefficient $\hat{\beta}_m(z)$ is estimated locally using ridge regression on the labeled subset to stabilize inversion when the local covariance is ill-conditioned or the number of labeled items is small relative to the signal dimension. The estimator uses a local kernel weighting scheme to handle continuous, ordered discrete, and unordered categorical profile spaces.

Extensions:
The framework extends to direct paired model gaps (estimating $\theta_a(z) - \theta_b(z)$ directly using within-item dependence) and deployment-weighted scores (estimating performance under a target deployment distribution $Q$ via importance weighting).

3. Key Contributions

Theoretical Guarantees:
- Calibration-Free Identification: Proves that the estimator is unbiased for the conditional profile regardless of the calibration of the cheap signals, provided they are centered.
- Unbiasedness: Establishes unbiasedness for grouped profiles under random sampling.
- Local Oracle Optimality: Demonstrates that within the class of centered linear augmentations, the estimator achieves oracle optimality.
- First-Order Adaptivity: Proves that estimating the nuisance coefficient $\beta(z)$ from the same scarce labels does not degrade the first-order asymptotic efficiency.
Efficiency Gain Formula: Derives a local gain formula $Gain_m(z) = 1 / \{1 - (1-\pi)R^2_m(z)\}$ , where $\pi$ is the labeled fraction and $R^2_m(z)$ is the population local coefficient of determination. This characterizes how efficiency gains vary across the profile space based on the explanatory power of cheap signals.
Unified Framework: Provides a single estimator that handles continuous, ordered discrete, and categorical profile spaces without requiring separate tuning for each.

4. Empirical Results

The authors evaluated LACE on eight benchmarks (MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, ARC) across three models (Claude Haiku 3, Ministral 3B, Qwen3 32B) and three label budgets ( $n=50, 100, 200$ ).

Performance: LACE consistently outperformed all baselines, including standard gold-label smoothing, plug-in estimators (using judge scores as direct predictors), and existing conditional inference methods (Local PPI, StratPPI, PPCI).
Efficiency Gains: Across 72 benchmark-model-budget cells, LACE achieved an overall geometric-mean Relative Efficiency (RE) of 5.528 (95% CI: [5.181, 5.898]) compared to the standard estimator. This implies that LACE achieves the same precision as the standard method with roughly 1/5.5th the number of gold labels.
Robustness: The method remained robust even when using weaker judge models (Llama-3.1-8B, GPT-OSS-20B) or when signals were noisy. Ablation studies confirmed that the gain is distributed across signal families but driven strongly by pairwise comparisons.
Simulation Validation: Controlled simulations verified the theoretical gain formula and demonstrated that LACE maintains superiority even when the useful signal direction changes across subgroups, a scenario where global methods fail.

5. Significance and Claims

The paper claims that cheap auxiliary signals need not be calibrated to be useful; they only need to explain local variation in gold correctness. LACE transforms this observation into a practical, semi-supervised estimator that:

Decouples Identification from Efficiency: It separates the estimation of the conditional mean from the variance reduction step, allowing the use of biased or miscalibrated signals.
Enables Fine-Grained Diagnosis: It makes it feasible to estimate performance profiles with very small label budgets, supporting better model selection, deployment targeting, and scientific diagnosis.
Generalizes Beyond LLMs: The authors note the scope extends to any setting with scarce gold outcomes, abundant cheap proxies, and a metadata covariate defining the conditional structure (e.g., clinical prediction with short-term biomarkers, online experimentation with engagement proxies).

The authors remain modest about limitations, noting that LACE improves efficiency only when cheap signals have local explanatory power (high local $R^2$ ) and that finite-sample covariance estimation can be unstable in small subgroups relative to signal dimension, though ridge regularization mitigates this. They do not claim the method solves the "curse of dimensionality" for high-dimensional profile spaces directly.