💬 NLP

Stopping and Routing LLM Judge Panels

This paper proposes a role-conditioned allocation framework that optimizes LLM judge panel construction by identifying judge roles (copies, complements, and specialists) to dynamically route, select, and stop calls, thereby generating an auditable and cost-effective evaluation strategy across diverse tasks.

Original authors: Bin Zhu, Yi Xie, Yanghui Rao

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bin Zhu, Yi Xie, Yanghui Rao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers have reached a point where the machines that write code, solve math problems, and generate stories are often judged by other machines. This practice, known as "LLM-as-a-judge," has become a standard way to evaluate new models. Instead of relying solely on human reviewers, who are expensive and slow, developers use large language models to act as referees, checking if an answer is safe, correct, or helpful. However, a new problem has emerged: there is no single perfect referee. Some models are excellent at spotting safety violations but poor at checking math. Others are great at general reasoning but fail on specific types of errors. Consequently, evaluation pipelines often involve a panel of different judges, each with a different specialty. The challenge for researchers is not just finding the best judge, but figuring out which judges to call, when to call them, and when to stop calling them altogether to save time and money.

A team of researchers at Sun Yat-sen University has tackled this logistical puzzle by treating the selection of judges not as a static list of the "best" tools, but as a dynamic decision-making process. They developed a method that analyzes a small set of test data to determine the specific role each potential judge plays in relation to the others. They found that judges generally fall into three categories: some are redundant copies that add no new information once a specific judge is already working; some are broad complements that improve the overall accuracy for every single example; and some are specialists that are only useful for a specific type of problem, such as detecting safety risks or handling difficult math.

The researchers' approach, which they call role-conditioned allocation, uses this understanding to build a custom plan for every batch of evaluations. Instead of blindly asking every available judge to review every single item, the system first checks if a judge is a copy. If a new judge provides no extra value beyond what the current team already knows, the system drops it entirely. If a judge offers broad improvements, it is added to the main team for all cases. If a judge is a specialist, the system routes it only to the specific types of examples where it is needed, such as sending a safety expert only to potentially dangerous responses. The process stops automatically when adding another judge no longer provides enough improvement to justify the cost.

To test this idea, the team applied their method to a wide variety of difficult tasks, including checking the logic of math solutions, verifying code against hidden tests, and evaluating the safety of chatbot responses. They compared their smart, selective approach against several other strategies: using just the single best judge, calling every judge on every task, or using simple rules based on confidence levels. The results showed that their method consistently found the right balance. On tasks like checking math reasoning, the system successfully combined a cheap, fast verifier with a few specialized language models, achieving high accuracy while using far fewer resources than calling everyone. On safety evaluations, the system learned to route specific judges only to risky cases, catching errors that a single general judge would miss, without wasting money on safe, straightforward examples.

The study also revealed when it is better to stop early. In cases where a simple, deterministic checker could solve the problem perfectly, the system correctly identified that no further judges were needed, preventing unnecessary spending. Conversely, for complex preference tasks where many different opinions might be valuable, the method knew when to keep the full panel active. By mapping out these different scenarios, the researchers provided a clear guide for developers: use a broad team when the task is complex and diverse, route specialists when the task has specific failure modes, and stop immediately when a simple tool has already solved the problem.

This work shifts the focus from simply counting how many judges are available to understanding how they work together. The researchers demonstrated that having a larger panel of judges does not automatically mean better results if the judges are redundant or called at the wrong time. By treating the evaluation process as a series of conditional decisions, they created a reusable plan that tells researchers exactly which judges to hire for the next batch of work. The outcome is a more efficient, transparent, and cost-effective way to ensure that artificial intelligence systems are being evaluated correctly, ensuring that resources are spent only where they make a measurable difference.

Technical Summary: Stopping and Routing LLM Judge Panels

1. Problem Statement

Large Language Model (LLM) evaluation pipelines increasingly rely on diverse "judge" candidates, including general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers. A critical deployment challenge arises: researchers must decide not only which judge is best, but which judges to call, on which examples, and when to stop constructing a judge panel.

Static rankings of judges fail to address this because a judge's value is conditional on the current panel composition, the target distribution, and the specific slice of examples being evaluated. Existing approaches often default to calling all judges (high cost), selecting a single best judge (potentially missing conditional value), or using heuristic diversity measures that do not account for target-relative information gain. The paper posits that a small labeled audit set can transform judge diversity from a descriptive property into a concrete calling policy.

2. Methodology: Role-Conditioned Allocation

The authors formulate judge-panel design as a role-conditioned allocation problem. The method estimates target-relative roles for candidate judges based on a small labeled audit set, declared slices (e.g., safety failures, adversarial subsets), and known judge costs.

2.1 Theoretical Framework

Let $X$ be an evaluated item and $Y \in [0, 1]$ be the audit label. For a panel $S$ , the oracle predictor $\eta_{P,S}$ estimates the target label given the joint output pattern $Z_S$ . The value of adding a candidate judge $j \notin S$ is defined as the reduction in squared-loss oracle risk:
$g_P(j | S) = R^\star_{P,S} - R^\star_{P,S \cup \{j\}}$
This gain represents the target information in $j$ not already present in $S$ . The authors establish a Projection Gain Identity (Lemma 1), showing this gain is non-negative and requires no independence assumptions among judges.

2.2 Role Taxonomy

Based on the conditional gain, candidates are classified into three roles:

Copy: The judge adds no conditional information beyond the current panel (gain $\approx 0$ ).
Complement: The judge improves the global panel across the target distribution (broad gain is high).
Specialist: The judge provides significant value only on specific slices (slice gain is high, but broad gain may be low).

A judge can simultaneously be a complement and a specialist. The method also tracks a specialization ratio to identify concentration of value on specific slices.

2.3 Construction Algorithm

The policy $\pi(x)$ is constructed via a greedy validation procedure:

Global Construction: Starting from an empty or seeded panel, the algorithm iteratively adds the candidate with the highest cost-adjusted validation gain, provided it exceeds a threshold $\tau_P$ .
Slice Routing: Once the global panel stops, the algorithm runs a similar greedy search for each declared slice $f$ . A specialist is added to a slice if its cost-adjusted slice gain exceeds a slice-specific threshold $\tau_f$ .
Stopping: Construction stops when no remaining candidate clears the respective threshold. The output is a "stopping report" documenting that unused candidates failed to justify their cost under the audit split.
Deployment: The final policy routes examples to the global panel $S$ , or to $S \cup S_f$ if the example belongs to slice $f$ . Route signals must be computable before the routed judge call (e.g., metadata, verifier outputs, classifier outputs), not ground-truth labels.

3. Key Contributions

Target-Relative Diversity: The paper argues that judge diversity should be defined relative to the target distribution and current panel, rather than as a nominal property. A judge may be a "copy" in one context and a "specialist" in another.
Practical Policy with Stopping: The authors convert the copy/complement/specialist taxonomy into a deployment policy that includes explicit stopping conditions. This allows researchers to determine when to drop redundant judges, when to route specialists, and when to stop early.
Regime Map for Deployment: The empirical results provide a "regime map" identifying when different strategies are optimal:
- Route specialists on deployable slices (e.g., safety proxies, adversarial subsets).
- Stop early in saturated verifier regimes (e.g., unit tests, answer checkers).
- Keep broad ensembles when the risk benefit outweighs the cost (e.g., complex preference modeling).
- Ignore conditional copies even if they increase nominal model diversity.

4. Experimental Results

The method was evaluated across seven diverse audit settings: Hard GSM8K rationale, MBPP public-overfit, JailbreakBench (JBB), LLMBar, RewardBench, Arena100K, SummEval, and MATH-500.

4.1 Performance vs. Baselines

The "Role Routed Stop" policy was compared against:

Single Best: The top-performing individual judge.
Flat All: Calling every judge.
Full-Call Stacking/Reliability Juries: Aggregating all outputs.
Frugal Cascades: Confidence-based routing.
Matched Diversity: Panels selected by quality or correlation diversity without role conditioning.

Key Findings:

Conditional Routing: On LLMBar (preference) and JBB (safety), the role policy significantly outperformed flat panels and single judges. For example, on LLMBar, it achieved 0.7334 accuracy at 3.46 calls, compared to 0.6692 for a flat panel at 7 calls.
Stopping Efficiency: On MBPP (code overfit) and Hard GSM8K, the policy combined a cheap verifier with a small number of LLM judges, achieving high accuracy (0.9900 on MBPP) at a cost of ~1.5 calls, far cheaper than full panels.
Saturated Regimes: On HumanEval and ordinary GSM8K, where verifiers saturate the target, the method correctly stopped after the verifier, avoiding unnecessary LLM calls.
Broad Ensembles: On RewardBench and MATH-500, the method identified that full-call aggregation remained the lowest-risk endpoint, correctly advising researchers to pay for the full panel in these regimes.

4.2 Stress Tests

Copy Stress Test: Adding four exact copies of a judge to the pool did not change the role policy's output (risk/cost remained stable), whereas full-call baselines saw increased cost and degraded performance due to over-weighting redundant votes.
Threshold Sensitivity: Varying the validation threshold $\tau$ allowed researchers to trade off between call cost and risk, demonstrating the policy's adaptability to different budget constraints.

5. Significance and Claims

The paper claims that its primary contribution is shifting the focus from describing judge diversity to deciding judge calls.

Operational Audit: The method produces a reusable, auditable call plan. The "stopping report" is operational, stating that under the specific audit split, costs, and thresholds, no further single judge calls are justified.
Deployment Reality: The authors emphasize that human labels are used only for audit analysis; deployment routes must rely on pre-available signals (metadata, verifiers, classifiers). The method successfully navigates this constraint.
Modest Scope: The paper does not claim to solve all evaluation problems or create new benchmarks. Instead, it offers a framework for making deployment decisions given a set of available judges and a labeled audit set. It highlights that a larger panel is not automatically more reliable, nor is a smaller panel automatically more efficient if it discards conditional information.

In conclusion, the paper provides a "regime map" for LLM judge allocation, enabling researchers to dynamically construct panels that balance cost, risk, and conditional information gain, rather than relying on static, one-size-fits-all evaluation strategies.