💬 NLP

SCOPE: Selective Conformal Optimized Pairwise LLM Judging

The paper proposes SCOPE, a framework that combines a novel Bidirectional Preference Entropy (BPE) uncertainty signal with selective conformal prediction to calibrate LLM pairwise judges, achieving reliable error control and significantly higher judgment coverage compared to standard baselines.

Original authors: Sher Badshah, Ali Emami, Hassan Sajjad

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sher Badshah, Ali Emami, Hassan Sajjad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive talent show where thousands of contestants (AI models) are competing. To decide who wins, you hire a "Judge" (another AI) to compare pairs of contestants and pick the better one.

The problem? This Judge isn't perfect. Sometimes it gets confused, sometimes it has a weird bias (like preferring the contestant who speaks first), and sometimes it is just guessing but acts like it's 100% sure. If you blindly trust every decision this Judge makes, your final rankings could be completely wrong.

This paper introduces SCOPE, a new system to make sure this AI Judge is reliable. Think of SCOPE as a safety inspector and a smart filter that sits between the Judge and the final results.

Here is how it works, broken down into three simple parts:

1. The Problem: The "Confident but Wrong" Judge

Usually, when an AI makes a choice, it gives a "confidence score" (like saying, "I'm 90% sure A is better than B"). But the paper found that this score is often a liar.

The Bias Trap: If you show the Judge "Contestant A then Contestant B," it might pick A. If you swap them to "B then A," it might pick B. The Judge is confused by the order of the presentation, not the actual quality.
The False Confidence: The Judge might say, "I'm 99% sure!" even when it's actually just guessing because of that order bias.

2. The Solution Part A: The "Double-Check" (BPE)

To fix the lying confidence, the authors created a new tool called BPE (Bidirectional Preference Entropy).

The Analogy: Imagine you are asking a friend, "Who is better, Alice or Bob?"
- Normal way: You ask once. Your friend says, "Alice!"
- The BPE way: You ask the same question twice, but you swap the order. First, "Alice vs. Bob." Then, "Bob vs. Alice."
- If your friend says "Alice" both times, they are truly confident.
- If they say "Alice" the first time and "Bob" the second time, they are confused.
The Score: BPE takes these two answers and calculates a "Confusion Score." If the Judge is confused (high score), BPE flags it as risky. If the Judge agrees with itself (low score), BPE says it's safe. This removes the bias of who was listed first.

3. The Solution Part B: The "Risk Budget" (SCOPE)

Now that we have a honest "Confusion Score," we need a rule for when to trust the Judge. This is where SCOPE comes in.

The Analogy: Think of a Risk Budget. You tell the system: "I am willing to accept a 10% error rate." (Meaning, out of every 100 decisions we keep, we are okay with up to 10 being wrong).
The Filter: SCOPE looks at the Confusion Score from the BPE step.
- If the score is low (Judge is clear and consistent), SCOPE says: "Keep this decision."
- If the score is high (Judge is confused or biased), SCOPE says: "Stop! Don't use this decision. We don't know enough."
The Guarantee: The paper proves mathematically that if you follow this rule, you will never exceed your 10% error budget. It's like a speed limit sign that guarantees you won't drive faster than the limit, no matter how the road conditions change.

Why is this a big deal?

Before this, people had to choose between two bad options:

Trust everything: Get lots of data, but it might be full of errors.
Trust nothing: Be super safe, but throw away 90% of the data because you aren't sure.

SCOPE is the sweet spot. It uses the "Double-Check" (BPE) to find the truly confident moments and the "Risk Budget" (SCOPE) to let you keep 2.4 times more correct decisions than before, while still guaranteeing you stay within your safety limits.

In short: SCOPE is a system that makes AI Judges honest about when they are confused, and it automatically filters out the risky guesses so you can trust the results you do keep.

Technical Summary: SCOPE (Selective Conformal Optimized Pairwise LLM Judging)

Problem Statement

Large Language Models (LLMs) are increasingly deployed as scalable "judges" for pairwise evaluation in AI workflows, ranging from Reinforcement Learning from Human Feedback (RLHF) to automated leaderboards. However, these models suffer from systematic miscalibration and biases, such as position bias (preferring the first or second response regardless of quality) and verbosity bias. Current methods for assessing judge reliability rely on heuristic confidence scores (e.g., maximum softmax probability) or simple thresholding, which lack finite-sample statistical guarantees. Consequently, it is difficult to ensure that the error rate among accepted judgments remains below a user-specified level, leading to distorted rankings and unreliable training signals.

Methodology

The paper proposes SCOPE (Selective Conformal Optimized Pairwise Evaluation), a framework that combines conformal risk control with a novel uncertainty estimator to provide rigorous statistical guarantees for pairwise LLM judging.

1. Bidirectional Preference Entropy (BPE)

To address the issue that standard confidence proxies are contaminated by position bias, the authors introduce Bidirectional Preference Entropy (BPE).

Mechanism: For a given pair of responses $(r_A, r_B)$ , the judge is queried twice: once in the forward order $(r_A, r_B)$ and once in the reverse order $(r_B, r_A)$ .
Aggregation: The model's preference probabilities for $r_A$ in both orders are averaged to create a bias-neutral probability $\bar{p}$ .
Scoring: The uncertainty score $s(x)$ is defined as the binary entropy of $\bar{p}$ :
$s(x) = -[\bar{p} \ln \bar{p} + (1 - \bar{p}) \ln (1 - \bar{p})]$
Properties: This score is permutation-invariant. Low entropy indicates high confidence (strong agreement across orders), while high entropy indicates ambiguity or disagreement between orders, serving as a robust signal for abstention.

2. Conformal Risk Control

SCOPE utilizes the theory of conformal prediction to calibrate an acceptance threshold $\hat{\lambda}$ that guarantees the False Discovery Rate (FDR) among accepted judgments does not exceed a user-specified level $\alpha$ .

Linearized Loss: To handle the ratio constraint of FDR control on finite samples, the method employs a linearized loss function $L(x, \lambda) = S(x, \lambda) \cdot (E(x) - \alpha)$ , where $S$ is the selection indicator and $E$ is the error indicator.
Calibration: The threshold $\hat{\lambda}$ is selected as the largest value such that the cumulative linearized loss on a calibration set satisfies a finite-sample sufficient condition: $\sum L(x_i, \lambda) \leq -1$ .
Guarantee: Under the assumption of exchangeability between calibration and test data, this procedure guarantees that the marginal test-time FDR is bounded by $\alpha$ .

Key Contributions

SCOPE Framework: A selective prediction method for pairwise LLM judging that provides finite-sample statistical guarantees on the error rate of accepted judgments, a capability previously missing in heuristic approaches.
Bidirectional Preference Entropy (BPE): A novel uncertainty estimator that mitigates position bias by aggregating predictions from both response orderings. BPE outperforms standard confidence proxies (e.g., predictive probability, verbalized confidence) in calibration and discrimination.
Empirical Validation: Extensive experiments across multiple benchmarks (MT-Bench, RewardBench, Chatbot Arena) and model scales (Qwen-2.5 series, Llama-3.1-70B) demonstrate that SCOPE consistently satisfies the target risk bound while maximizing coverage.

Results

Uncertainty Quality: BPE achieves lower Expected Calibration Error (ECE) and higher Area Under the ROC Curve (AUROC) and Precision-Recall Curve (AUPRC) compared to baselines, including "Swap-and-Aggregate" and "Simulated Annotators."
Risk Control: Across all benchmarks and models, SCOPE maintains empirical FDR tightly concentrated around the target $\alpha$ (e.g., $\approx 0.097\text{--}0.099$ at $\alpha = 0.10$ ). In contrast, vanilla baselines and heuristic thresholding frequently violate the risk constraint.
Coverage Efficiency: SCOPE significantly outperforms conservative baselines. At $\alpha = 0.10$ , SCOPE accepts up to 2.4 $\times$ more judgments than naive calibration methods while maintaining the same risk bound. For instance, on MT-Bench with Qwen-7B, SCOPE achieves 0.246 coverage compared to 0.102 for naive calibration.
Robustness: The framework remains valid under label noise in the calibration set (becoming more conservative rather than violating the bound) and requires modest calibration set sizes ( $n \approx 200\text{--}500$ ) to converge.

Significance and Claims

The paper claims that SCOPE provides the first framework to offer finite-sample false-discovery-rate control for single-model pairwise LLM judging. By decoupling the uncertainty signal (BPE) from the risk control mechanism (Conformal Prediction), the authors demonstrate that it is possible to deploy LLM judges as scalable evaluators without sacrificing trustworthiness.

The authors position SCOPE as a "drop-in" calibration layer that requires no retraining of the underlying judge model. It transforms LLM judging from an opaque black box into a system with known, user-tunable reliability properties. The work suggests that combining bias-neutral uncertainty estimation with conformal risk control is a foundational step toward accountable and transparent AI evaluation, particularly for applications like RLHF reward modeling and leaderboard rankings where systematic errors can have downstream consequences.

The paper acknowledges limitations, noting that guarantees rely on the exchangeability assumption (which may be weakened by distribution shifts) and that BPE requires access to model logits (making it less applicable to fully black-box API-only settings without approximation). Future work is suggested for extending these guarantees to multi-response ranking and rubric-based scoring.