💬 NLP

Mediocrity is the key for LLM as a Judge Anchor Selection

This paper demonstrates that the selection of an anchor model is critical for the reliability of "LLM-as-a-judge" evaluations, revealing that extreme anchors (best or worst models) undermine ranking accuracy and advocating for the use of "mediocre" anchors alongside larger benchmark sizes to ensure robust results.

Original authors: Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why "Average" is the New "Best"

Imagine you are a talent scout trying to rank 22 singers in a competition. You can't listen to every single pair of singers against each other (that would take forever and cost a fortune). So, you decide to pick one singer to be the "Anchor." You will only compare every other singer against this one Anchor.

The common wisdom in the AI world has been: "Pick the best singer as your Anchor!" or "Pick the worst singer as your Anchor!" The idea was that if you compare everyone to the absolute best, you'll see who is close to the top. If you compare everyone to the worst, you'll see who is better than the bottom.

This paper says: You are wrong. Both of those choices are terrible.

Instead, the paper argues that the "mediocre" singer—the one who is just okay, right in the middle of the pack—is actually the perfect Anchor.

The Problem: The "Superstar" and the "Struggling" Anchor

Let's look at why picking the extremes fails, using two analogies:

1. The Superstar Anchor (The "Olympian")

Imagine your Anchor is a world-famous Olympic gold medalist.

The Scenario: You ask the judge to compare a local high school singer against the Olympian.
The Result: The judge immediately says, "The Olympian wins!" 99 times out of 100.
The Problem: This gives you zero useful information. You already knew the Olympian was better. You wasted your time and money on a comparison that didn't tell you anything about the other singers. You can't tell if Singer A is slightly better than Singer B because both of them lost to the Olympian in a landslide.

2. The Struggling Anchor (The "Amateur")

Now, imagine your Anchor is someone who can barely carry a tune.

The Scenario: You compare a professional singer against this amateur.
The Result: The judge immediately says, "The pro wins!" 99 times out of 100.
The Problem: Again, zero useful information. Everyone beats the amateur. You still don't know who is the best among the professionals.

The Conclusion: When you pick an Anchor that is too good or too bad, the results are "saturated." It's like trying to measure the height of a skyscraper and a garden gnome using a ruler that only goes up to 10 feet. You can't distinguish between the tall things, and you can't distinguish between the short things.

The Solution: The "Goldilocks" Anchor

The paper found that the best Anchor is the mediocre one—the singer who is average.

Why it works: When you compare a good singer to an average Anchor, the judge has to think hard. "Hmm, this one is slightly better."
The Result: When you compare a bad singer to the same average Anchor, the judge also has to think. "Hmm, this one is slightly worse."
The Magic: Because the Anchor is in the middle, the results are balanced. Some people win, some people lose, and the margins are close. This creates a rich dataset where you can actually see the subtle differences between the singers.

The Analogy: Think of the Anchor as a thermostat.

If the thermostat is set to 100°F (too hot), everyone feels cold. You can't tell who is shivering a little and who is shivering a lot.
If the thermostat is set to 32°F (too cold), everyone feels hot. Same problem.
If the thermostat is set to a comfortable 70°F, you can clearly feel who is slightly too warm and who is slightly too cool. That "Goldilocks" temperature gives you the most data.

The Surprising Findings

The researchers did a massive experiment (comparing over 850,000 pairs of AI models) and found three shocking things:

The "Inverted U" Shape: If you plot the "quality" of the Anchor against how well it ranks the other models, you get a U-shape. The best Anchors are in the middle (the bottom of the U), and the worst Anchors are at the very top and very bottom.
It Matters More Than the Judge: People spend a lot of time arguing about which AI model should do the judging (e.g., "Is GPT-4 better than Claude?"). The paper found that choosing the right Anchor is just as important as choosing the right Judge. A bad Anchor can ruin your results even if you have the smartest Judge in the world.
We Are Wasting Money: Current benchmarks (like Arena-Hard) are often too small. Because "extreme" Anchors waste so much data (by producing too many obvious wins/losses), we need much larger datasets to get reliable results. If you use a bad Anchor, you might need double the data to get the same accuracy.

What Should You Do? (The Cheat Sheet)

If you are running an AI evaluation, here is the advice from the paper:

Don't pick the "Best" model as your Anchor. It's too easy to beat.
Don't pick the "Worst" model. It's too easy to lose to.
Pick the "Middle" model. Find a model that is roughly as smart as the ones you are testing. This will give you the most accurate ranking.
Check your data first. Before running a huge, expensive test, run a tiny test to see if your Anchor is "informative" (does it produce a mix of wins and losses?). If it's too one-sided, pick a different Anchor.

The Takeaway

In the world of AI evaluation, mediocrity is a superpower. By choosing a "good enough" Anchor instead of a "perfect" or "terrible" one, you get a clearer, fairer, and more accurate picture of how AI models actually compare to each other. It's the difference between a blurry photo and a high-definition picture.

1. Problem Statement

The "LLM-as-a-Judge" (LMJ) paradigm has become the standard for evaluating open-ended text generation, replacing traditional reference-based metrics. To address the quadratic computational cost of pairwise comparisons (where every model is compared against every other model), popular benchmarks like Arena-Hard and AlpacaEval utilize an anchor-based evaluation strategy. In this approach, all models are compared against a single "anchor" model to derive a ranking.

However, the selection of this anchor model has been largely treated as a heuristic or a static default (often choosing the strongest or weakest available model). The paper identifies a critical gap: the impact of anchor selection on the reliability and validity of the resulting rankings is largely unexplored. The authors hypothesize that poor anchor choices may lead to noisy rankings that fail to correlate with human preferences or ground-truth rankings.

2. Methodology

The authors conducted a large-scale empirical study to systematically investigate the effect of anchor selection.

Dataset & Scope:
- Benchmark: Arena-Hard-v2.0 (750 challenging queries) and AlpacaEval (805 instructions).
- Models: 22 contemporary LLMs (including o1, o3, GPT-4.5, Llama 4, etc.) serving as both candidates and potential anchors.
- Judges: 5 different LLM judges (Deepseek-v3, GPT-OSS 120B/20B, Qwen3 235B/8B).
Experimental Design:
- Pairwise vs. Anchor: The authors generated a "Gold Standard" ranking ( $\pi_{quad}$ ) by performing all possible pairwise comparisons ( $\binom{22}{2} \times 750 \approx 173k$ comparisons per judge).
- Anchor Simulation: They evaluated the Arena-Hard dataset using each of the 22 models as the anchor individually against the other 21 models.
- Metrics:
  - Ranking Correlation: Kendall's $\tau$ correlation between the anchor-based ranking and the Gold Standard ( $\pi_{quad}$ ) and Human Ranking ( $\pi_{human}$ ).
  - Informativeness ( $I(p, A)$ ): Defined as the fraction of samples where the anchor's verdict distinguishes between two competing models (i.e., the anchor does not tie with both or beat/lose both in a way that provides no relative signal).
Statistical Analysis:
- Power Analysis: Calculated the required sample size ( $N$ ) to detect a small effect size (+5% win rate difference) given the informativeness rates of different anchors.
- Robustness Checks: Tested sensitivity to dataset size and the impact of using multiple anchors.

3. Key Findings & Results

A. The "Mediocrity" Phenomenon (Inverted U-Shape)

The most significant finding is an inverted U-shaped relationship between an anchor's capability rank and its effectiveness as an anchor.

Poor Anchors: The strongest (top-ranked) and weakest (bottom-ranked) models make the worst anchors.
- Example: Using o3 (the strongest model) as an anchor resulted in a Kendall's $\tau$ of only 0.818 against the quadratic ranking, whereas the best anchor achieved 0.957.
- Reasoning: Strong anchors win almost all comparisons (signal saturation), while weak anchors lose almost all. This results in a skewed distribution of win-rates where the majority of samples provide no discriminative information about the relative ordering of the other models.
Optimal Anchors: "Mediocre" models (those in the middle of the capability distribution) yield the highest correlation with the ground truth. They produce a flatter, more evenly distributed win-rate curve, maximizing the number of informative samples.

B. Informativeness and Sample Efficiency

Wasted Budget: Using a top-tier anchor like o3 resulted in only 45% informative samples (55% of the compute budget was wasted on ties or trivial wins/losses).
Sample Size Requirements: Due to low informativeness, standard benchmark sizes (e.g., 750 samples) are statistically insufficient to distinguish between competitive models when using a poor anchor.
- To detect a +5% effect size with a poor anchor (45% informativeness), the study estimates a need for ~1,372 samples, far exceeding the 750 samples in Arena-Hard.
- Even with the best anchors, ~39% of samples are ties, indicating inherent limitations in anchor-based evaluation.

C. Anchor Selection vs. Judge Selection

The authors compared the magnitude of the impact of selecting an anchor versus selecting a judge.

Result: The choice of anchor has a comparable or even larger impact on evaluation reliability than the choice of the judge model.
Data: The gap in correlation between the best and worst anchors (up to 0.19 drop) is significant and often exceeds the variance introduced by switching between different judge models.

D. Predictive Estimation

The study demonstrated that anchor informativeness can be reliably estimated using a small subset of data (e.g., 10 samples). This allows practitioners to identify "mediocre" anchors before running the full evaluation, avoiding the computational cost of testing all candidates.

4. Key Contributions

Systematic Empirical Analysis: The first large-scale study (850k+ comparisons) quantifying the effect of anchor selection on LMJ reliability.
Discovery of the "Mediocrity" Principle: Establishing that mid-tier models are superior anchors compared to state-of-the-art or weak baselines, challenging the common practice of using "Gold Standard" strong models as anchors.
Statistical Power Analysis: Demonstrating that current benchmark sizes are often insufficient for anchor-based evaluation due to high rates of non-informative samples (ties/saturation).
Actionable Guidelines: Providing a decision framework (Fig. 5 in the paper) for when to use anchor-based evaluation and how to select anchors.

5. Significance and Recommendations

This paper fundamentally challenges the standard operating procedure of LLM leaderboards and benchmarks.

For Benchmark Creators: Avoid using the "best" model as an anchor. Instead, select a model with capability similar to the median of the models being evaluated to maximize statistical power.
For Evaluation Design:
- Avoid Anchor-Based Evaluation for small model sets ( $N \le 3$ ) or when a natural baseline exists; use full pairwise comparisons instead.
- Report Informativeness: Papers using anchor-based evaluation must explicitly report the anchor's informativeness rate to validate the results.
- Sample Size: Increase dataset sizes significantly if anchor-based evaluation is unavoidable, or use weighted statistical tests (e.g., Wilcoxon signed-rank) to leverage magnitude information.
Broader Impact: The findings suggest that the reliability of LLM leaderboards (like Chatbot Arena) may be compromised if anchors are chosen poorly, potentially misranking competitive models. The "mediocrity" principle offers a simple, low-cost method to improve the fidelity of automated evaluations.

In conclusion, the paper argues that anchor selection is a critical hyperparameter in LMJ evaluation, as important as the judge model itself, and that "mediocre" anchors are the key to unlocking reliable, efficient, and statistically sound model rankings.