💬 NLP

Localize-Then-Decide Guarantees for LLM Judgments

This paper proposes a "Localize-Then-Decide" framework that combines conformal prediction to generate a high-probability shortlist with a calibrated confidence-based selection rule, thereby restoring the reliability of confidence estimates and enabling robust guarantees for LLM judgments across varying candidate sizes.

Original authors: Xinyu Li, Yi Zhou, Guanqun Cao, Zeyu Fu, Tianjin Huang, Gaojie Jin

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinyu Li, Yi Zhou, Guanqun Cao, Zeyu Fu, Tianjin Huang, Gaojie Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving landscape of artificial intelligence, large language models have become powerful tools not just for writing or chatting, but for judging the quality of other machines. When a computer program generates several different answers to a single question, a human or another computer model must decide which one is best. This process is increasingly automated, with AI systems acting as referees to evaluate everything from creative writing to safety guidelines. However, a fundamental problem remains: how can we trust these digital judges? When a model claims to be highly confident in its choice, does that confidence actually mean it is right? For years, researchers have relied on the assumption that higher confidence scores naturally correlate with higher accuracy, a belief that works well when a model is simply choosing between two options. But as the number of choices grows, this simple logic begins to falter, creating a blind spot in our ability to verify automated decisions.

A team of researchers has identified exactly where this logic breaks down and proposed a new method to fix it. They found that when an AI judge is forced to pick the best response from a large group of candidates—say, twenty different answers instead of just two—the math behind its confidence becomes distorted. Imagine a pie representing the total certainty the model has; when there are only two slices, the best slice is large and easy to identify. But as the number of slices increases to twenty, the pie must be divided among all of them, making even the best slice appear small. This dilution causes the model's confidence score to drop, even if it has correctly identified the best answer, breaking the reliable link between high confidence and high accuracy. Consequently, standard methods that try to filter out uncertain judgments by setting a confidence threshold fail, often rejecting good answers or accepting bad ones simply because the numbers look wrong.

To solve this, the researchers developed a two-step framework they call "Localize-Then-Decide." Instead of trying to pick the single best answer immediately from a crowded field, the system first narrows the field down to a very small, high-quality shortlist. In this first stage, the model uses a statistical technique to identify a small group of candidates that almost certainly contains the human-preferred answer. This step acts as a safety net, ensuring the true best option is not lost in the noise of the larger group. Once the field is reduced to just a few top contenders, the system moves to the second stage. Here, it attempts to select the single best response from that small shortlist. Because the competition is now limited to just a few options, the relationship between confidence and accuracy is restored; the model can once again reliably distinguish between a sure bet and a guess. If the model cannot find a clear winner even within this small group, it is designed to step back and admit uncertainty rather than making a risky guess.

The team tested this approach across multiple datasets and with various AI models, ranging from smaller systems to massive, complex ones. They compared their two-step method against traditional single-step approaches that try to pick a winner directly from the full list of candidates. The results were clear: the new framework consistently achieved much higher rates of success in meeting their safety targets. While the old methods often failed to provide reliable guarantees when the number of candidates increased, the new system maintained a high level of trustworthiness. For instance, in scenarios where the goal was to ensure the model agreed with human preferences at least 81 percent of the time, the traditional methods frequently fell short, achieving success rates as low as 50 percent in some cases. In contrast, the two-step approach consistently exceeded 90 percent success rates, effectively proving that the system could be trusted to make the right call when it said it was confident.

Furthermore, the researchers demonstrated that this method could be scaled up into a cascading system, where different models work together to handle the workload efficiently. In this setup, a smaller, faster model would first try to localize the best answers. If it felt uncertain, the task would be passed to a larger, more powerful model, which would then apply the same two-step logic. This architecture allowed the system to handle difficult cases with high reliability while saving computational resources on easier ones. The study confirms that by breaking a complex decision into a localization phase followed by a selection phase, we can restore the mathematical guarantees needed for trustworthy automated evaluation. This work does not just improve how we measure AI performance; it provides a structural solution to a problem that has long plagued the field, ensuring that as AI judges become more common, their confidence remains a true reflection of their accuracy.

Technical Summary: Localize-Then-Decide Guarantees for LLM Judgments

Problem Statement

Large Language Models (LLMs) are increasingly deployed as evaluators ("LLM-as-a-judge") to assess output quality and preference alignment. While recent work (e.g., Jung et al., 2024) has established methods to provide statistical guarantees for human-LLM agreement in pairwise settings ( $m=2$ ), these methods face a fundamental breakdown in multi-candidate settings ( $m > 2$ ), which are common in practical applications like best-of- $N$ decoding and multi-system comparisons.

The core issue is the violation of the monotonicity assumption: the premise that higher estimated confidence implies lower disagreement risk with human judgments. In pairwise comparisons, this relationship often holds. However, as the number of candidates $m$ increases, the total probability mass must be distributed across more alternatives. This "structural dilution" causes the estimated confidence of the true best response to decrease, even if the model's relative preference remains stable. Consequently, the relationship between confidence scores and human-agreement rates becomes non-monotonic, rendering standard confidence-thresholding guarantees unreliable for $m > 2$ .

Methodology: Localize-Then-Decide Framework

To address the monotonicity breakdown, the authors propose a Localize-Then-Decide framework, which decomposes the $m$ -to-1 selection task into two calibrated stages. This decomposition restores the monotonic relationship between confidence and disagreement risk.

Stage I: Conformal Area Localization

The first stage employs conformal prediction to localize a small shortlist of candidates that contains the human-preferred response with high probability.

Scoring: Uses an Ensemble Mean Probability (EMP) score, $S_c(x, g_i)$ , derived from simulated annotators.
Nonconformity Score: Defined as the rank of the human-best response under $S_c$ .
Calibration: Using a calibration set, the method determines a conformal quantile $k$ such that the top- $k$ candidates form a set $S(x)$ satisfying:
$P(y \in S(x)) \geq 1 - \alpha$
where $\alpha$ is the user-specified risk tolerance. This step effectively filters the candidate space from $m$ to a small $k$ (e.g., $k=3$ ) without requiring the model to be certain of the single best answer.

Stage II: Selective Auto-Pick

The second stage attempts to select a single response from the localized shortlist $S(x)$ or abstain if reliability cannot be guaranteed.

Scoring: Uses a margin-based confidence score, $\Delta_{S(x)}$ , defined as the difference between the score of the predicted winner and the runner-up within $S(x)$ . The authors utilize a KL-divergence based scoring function with variance normalization to compute these scores.
Calibration: A fixed-sequence testing rule is applied to the calibration subset where the ground truth is covered by $S(x)$ . The system searches for the largest threshold $\lambda^*$ such that the empirical error rate on accepted instances remains below a target risk $\epsilon$ .
Decision: If the margin $\Delta_{S(x)} \geq \lambda^*$ , the system outputs the predicted winner; otherwise, it abstains.

Theoretical Guarantee

By combining the coverage guarantee of Stage I and the conditional error control of Stage II, the framework provides a composite guarantee. Under the assumption that the probability of missing the shortlist decreases monotonically with the Stage II threshold, the framework guarantees:
$P(\text{LLM agrees with human} \mid \text{Satisfied confidence}) \geq (1 - \epsilon)(1 - \alpha)$
with probability at least $1 - \delta$ over the calibration sample.

Key Contributions

Practical Extension to Multi-Candidate Settings: The work extends human-LLM judgment guarantees from the pairwise regime ( $m=2$ ) to the practical multi-generation regime ( $m > 2$ ). It identifies and characterizes the "probability dilution" effect that breaks monotonicity in existing single-stage approaches.
Two-Stage Guarantee Framework: The authors propose a novel two-stage architecture (Conformal Localization + Selective Auto-Pick). This design empirically and theoretically restores the monotonic confidence-agreement relationship, enabling valid statistical guarantees for $m > 2$ .
Cascaded Architecture Enablement: The framework naturally supports cascaded evaluation architectures, where weaker models handle easy cases and escalate difficult instances to stronger models. Because the framework provides valid per-tier guarantees, the composition of these tiers maintains overall reliability, unlike single-stage cascades which fail under $m > 2$ .

Experimental Results

The framework was evaluated across four datasets (TL;DR, Chatbot Arena, HH-RLHF, AlpacaEval) and multiple judge LLMs (ranging from 7B to 120B parameters) with candidate sizes $m \in \{5, 10, 20\}$ .

Restoration of Monotonicity: Experiments confirmed that single-stage ranking loss (measuring monotonicity violation) increases significantly with $m$ (e.g., rising from 0.14 to 0.34 for Qwen2.5-72B on TL;DR as $m$ goes from 5 to 20). In contrast, both Stage I (localization) and Stage II (selection within shortlist) maintained low ranking loss ( $\leq 0.08$ ), validating the restoration of monotonicity.
Guarantee Success Rate (GSR) and Coverage:
- Single-Stage Baselines: Failed to meet the target guarantee success rate (GSR) of $1-\delta = 90\%$ for most settings, often dropping to 50–70% as $m$ increased.
- Two-Stage Framework: Consistently achieved GSRs above 90% (often 93–97%) across all datasets and models.
- Coverage: The two-stage framework significantly improved coverage (the fraction of instances accepted) compared to single-stage baselines. For example, on TL;DR with GPT-OSS-120B, coverage increased from ~56% (single-stage) to ~70% (two-stage) while maintaining the guarantee.
Cascaded Evaluation: In cascaded setups (e.g., 7B $\to$ 32B $\to$ 72B), the two-stage approach achieved GSRs of 91–94% and coverage of 77–81%, whereas single-stage cascades failed to reach the 90% GSR threshold. The cascades also demonstrated computational efficiency, with the strongest model invoked for less than 30% of instances.

Significance and Claims

The paper claims that the Localize-Then-Decide framework is a critical step toward trustworthy and scalable "LLM-as-a-judge" systems. By decomposing the decision process, the authors demonstrate that it is possible to provide finite-sample, distribution-free guarantees for multi-candidate evaluation, a setting where previous confidence-thresholding methods were fundamentally unreliable.

The work emphasizes that the framework is modular and orthogonal to specific scoring functions, allowing it to integrate with various confidence estimators (e.g., Simulated Annotators, BPE). The authors position this as a solution to the "reliability gap" in automated evaluation pipelines, enabling the safe use of LLM judges in high-stakes scenarios where statistical assurance of alignment with human preferences is required. The paper notes limitations regarding the assumption of exchangeability between calibration and test data and the computational cost of confidence estimation, but asserts that the framework is broadly applicable across diverse evaluation domains.