💬 NLP

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

This paper proposes a risk-controlled framework for LLM judging that dynamically routes instances between parametric and retrieval-augmented modes based on calibrated uncertainty thresholds, thereby guaranteeing that the false discovery rate of accepted verdicts remains below a user-specified level while maximizing coverage.

Original authors: Sher Badshah, Ali Emami, Hassan Sajjad

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sher Badshah, Ali Emami, Hassan Sajjad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new standard has emerged for testing how well computer programs understand the world. Instead of relying on human experts to read and grade every answer a machine produces, researchers now often use one large language model to act as a judge for another. This approach is efficient and scalable, allowing developers to evaluate thousands of responses in the time it would take a human to grade a single essay. However, a significant problem arises when these digital judges are asked to verify facts. Unlike subjective tasks where there is no single right answer, factual questions have clear truths. A judge might confidently declare a false statement to be true simply because it sounds plausible or because the model has forgotten a recent event. Without a way to know when the judge is unsure, these silent errors can slip through, leading to unreliable conclusions.

The core challenge lies in balancing confidence with accuracy. If a judge is too cautious, it refuses to answer most questions, rendering the system useless. If it is too eager, it makes mistakes. Researchers have tried to solve this by having the judge admit when it is uncertain, but this often means discarding potentially correct answers that the model just didn't feel sure about. A new study published at the COLM 2026 conference proposes a smarter middle ground. The researchers, Sher Badshah, Ali Emami, and Hassan Sajjad, developed a system that allows the judge to pause and seek help before giving up. When the judge feels unsure about a fact based on its internal memory, it does not immediately refuse to answer. Instead, it is equipped with a tool to search the web for evidence. It then re-evaluates the question using this new information. Only if the judge remains uncertain even after reading the search results does it admit defeat and flag the question for human review.

The team tested this two-step approach on a variety of difficult question-answering benchmarks, using different sizes of language models to act as both the student and the judge. They found that by adding this retrieval step, the system could confidently answer many more questions than a judge working alone. Crucially, the researchers did not just hope this would work; they built a mathematical safety net around the process. They calibrated the system on a set of known questions to ensure that the rate of mistakes among the answers it accepted stayed below a specific, user-defined limit. For instance, if a user set a limit of five percent, the system was guaranteed to keep its error rate at or below that level with high statistical certainty. This guarantee held true even when the judge switched from using its own knowledge to using web search results, a complex transition that previous methods struggled to manage without losing control over the error rate.

The results showed that this adaptive strategy significantly improved the system's ability to provide answers without sacrificing reliability. On some of the harder datasets, the system was able to accept answers for nearly all the questions it was asked, whereas a judge working without web search would have refused to answer the vast majority of them. The study demonstrated that the system could recover coverage on difficult, knowledge-intensive tasks that would otherwise be lost to uncertainty. Furthermore, the researchers checked whether the system remained reliable even when the web search results changed over time, simulating a real-world scenario where information on the internet shifts. They found that the safety guarantees held firm, proving that the system could adapt to new information without needing to be completely re-calibrated every time the web changed.

This work offers a practical path forward for deploying artificial intelligence in situations where factual accuracy is paramount. By combining the speed of automated judging with the reliability of human-like verification through search, the system bridges the gap between efficiency and trust. It shows that machines can be taught to recognize their own limits, not just by staying silent, but by knowing when to look up the answer. The study concludes that with the right safeguards, we can build evaluation systems that are both bold enough to answer difficult questions and careful enough to ensure those answers are correct, providing a robust foundation for the next generation of AI applications.

Technical Summary: Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

Problem Statement

The use of Large Language Models (LLMs) as judges has become standard for evaluating model outputs at scale, particularly for subjective tasks. However, applying LLM judges to objective, factual tasks (e.g., open-domain QA) presents distinct reliability challenges. Without a reference answer, judges rely on parametric knowledge, which risks hallucination or lack of knowledge for recent/rare entities, or tool augmentation, which introduces computational costs and lacks a mechanism for reliable usage.

Crucially, existing uncertainty quantification methods are often heuristic; applying a threshold to uncertainty scores provides no formal guarantee on the error rate among accepted verdicts. Furthermore, current risk-controlled selective prediction methods are designed for filtering examinee answers in QA tasks rather than controlling errors in a meta-evaluation setting. Finally, existing approaches often treat "abstention" as the only recourse for uncertainty, discarding instances where the judge's uncertainty stems from a knowledge gap that could be resolved with additional evidence.

Methodology

The authors propose a risk-controlled selective evaluation framework that integrates adaptive retrieval with formal risk guarantees. The framework operates in two modes and utilizes a two-threshold routing mechanism calibrated on a held-out set with known ground-truth labels.

1. Problem Formulation

The goal is to select a threshold (or pair of thresholds) such that the False Discovery Rate (FDR)—the error rate among instances the judge chooses to evaluate—remains below a user-specified risk level $\alpha$ with high probability ( $1-\delta$ ).
$\text{Pr}(R(t) \leq \alpha) \geq 1 - \delta$
where $R(t)$ is the true FDR.

2. Two-Mode Routing Mechanism

The framework employs a hierarchical evaluation process:

Mode 1 (Parametric): The judge evaluates the candidate answer using only its internal knowledge, producing a verdict $v_1$ and an uncertainty score $U_1$ .
Mode 2 (Retrieval-Augmented): If $U_1$ exceeds the first calibrated threshold $\hat{t}_1$ , the system retrieves top- $k$ web results based on the question. The judge re-evaluates the instance with this evidence, producing $v_2$ and $U_2$ .
Abstention: If $U_2$ exceeds the second calibrated threshold $\hat{t}_2$ , the instance is abstained from (flagged for human review).

3. Calibration and Theoretical Guarantees

The thresholds are calibrated using Clopper–Pearson exact upper confidence bounds on a held-out calibration set.

Single-Mode: For a fixed threshold, the selection event is deterministic given the input, preserving the i.i.d. Bernoulli structure of failure indicators. This allows the application of the Clopper–Pearson bound to guarantee FDR control.
Two-Mode Extension: The authors prove that this Bernoulli structure extends to the two-threshold routing setting. Since the routing decision is a deterministic function of the uncertainty scores and fixed thresholds, the failure indicators of the selected subset remain i.i.d. Bernoulli. Consequently, the finite-sample FDR guarantee holds for the joint policy without additional distributional assumptions.

The calibration process searches for the threshold pair $(\hat{t}_1, \hat{t}_2)$ that maximizes coverage (the number of accepted instances) while ensuring the upper confidence bound of the empirical FDR remains $\leq \alpha$ .

Key Contributions

Risk-Controlled Selective Evaluation: The paper formulates risk-controlled selective evaluation specifically for the LLM-as-a-judge setting on factual QA.
Adaptive Retrieval Routing: It introduces a two-mode mechanism that equips the judge with adaptive web retrieval to resolve uncertainty before resorting to abstention, effectively turning potential abstentions into informed evaluations.
Theoretical Extension: It demonstrates that the Bernoulli structure required for the Clopper–Pearson bound extends to the two-threshold routing setting, providing a finite-sample FDR guarantee for the joint policy.
Empirical Validation: Experiments across open-domain QA benchmarks and varying judge scales show that the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines.

Results

The framework was evaluated on four datasets (TriviaQA, Natural Questions, HotpotQA, PopQA) using various candidate models (Qwen3-8B, LLaMA-3.1-70B) and judges (Qwen3-4B/8B/14B, LLaMA-3.1-8B).

FDR Control: Across all 32 configurations, the empirical FDR remained at or below the target $\alpha$ , satisfying the finite-sample guarantee. The FDR tracked closely to $\alpha$ with narrow confidence bands.
Coverage Gains: Adaptive retrieval significantly recovered coverage that would otherwise be lost to abstention.
- On knowledge-intensive tasks (e.g., HotpotQA, NQ-Open), Mode 2 (retrieval) accounted for the majority of non-abstained verdicts.
- For example, on NQ-Open with a Qwen3-8B candidate, coverage increased from 7% (Mode 1 only) to 82% (Joint framework) at $\alpha=0.20$ .
- On HotpotQA with LLaMA-8B, coverage rose from 40% to 85%.
Retrieval Impact: While retrieval expanded coverage, it did not always reduce uncertainty. For 14–29% of instances, retrieval increased uncertainty (e.g., due to conflicting sources). The second threshold successfully routed these cases to abstention, ensuring only high-confidence, evidence-grounded verdicts were accepted.
Robustness: The framework remained robust to retrieval drift (changes in web search results over time), with FDR control preserved when applying calibrated thresholds to fresh evidence snapshots.

Significance

The paper claims that its primary significance lies in providing formal, finite-sample risk guarantees for LLM judges in objective evaluation settings, a capability previously lacking in heuristic thresholding approaches. By combining adaptive retrieval with risk-controlled selection, the framework addresses the trade-off between reliability and coverage: it allows judges to leverage external tools to resolve knowledge gaps rather than simply abstaining, all while mathematically bounding the rate of incorrect accepted verdicts. This enables the deployment of LLM judges in high-stakes or autonomous agent scenarios where uncontrolled error propagation is unacceptable.