💬 NLP

Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks

This paper reveals that LLM-based judges exhibit significant intra-rater inconsistency across different runs, undermining the reliability of their scores, and investigates whether such evaluations can still be effectively utilized through specific guidelines.

Original authors: Rajarshi Haldar, Julia Hockenmaier

Published 2026-04-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rajarshi Haldar, Julia Hockenmaier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Rollercoaster" Judge

Imagine you hire a famous food critic to review a new restaurant. You ask them to taste the same dish three times in a row, sitting at the same table, with the same napkin in front of them.

Taste 1: They give it a 5-star rating, calling it "a masterpiece."
Taste 2: They give it a 2-star rating, calling it "bland and overcooked."
Taste 3: They give it a 4-star rating, saying it's "decent but needs more salt."

If this happened, you wouldn't trust that critic. You'd realize their judgment is like a roulette wheel—it spins and lands on a different number every time, even though the food (the input) never changed.

This is exactly what the researchers found when they asked Large Language Models (LLMs) to act as judges for other AI-generated text. They discovered that these AI judges are suffering from "Rating Roulette."

The Problem: The AI Judge Can't Agree with Itself

In the world of AI, we often use one AI to grade the work of another AI. This is called "LLM-as-a-Judge." It's popular because it's faster and cheaper than hiring humans.

However, the researchers ran a simple test:

They took a piece of text (like a news summary or a chat conversation).
They asked the same AI judge to grade it three times using the exact same instructions.
They checked if the AI gave the same score each time.

The Result: The AI judges were all over the place.

On a task called SummaC (checking if a summary is factually true), the best AI judge only agreed with itself about 79% of the time.
On a task called MT-Bench (ranking chatbot conversations), the agreement dropped even lower. One AI judge only gave the exact same answer three times in a row for just 61% of the cases.

It's as if the judge's mood changes every time you ask a question, even though the question and the answer haven't changed.

The "Magic Trick" That Backfires

You might think, "Okay, let's just tell the AI to stop being random. Let's turn off the 'dice rolling' part of its brain so it always gives the same answer."

The researchers tried this. They set the AI to be 100% deterministic (no randomness).

Did it become consistent? Yes.
Did it become a better judge? No.

In fact, when they forced the AI to stop rolling the dice, its grades actually became worse compared to human judges. It's like forcing a chef to use the exact same recipe every single time; they might stop making mistakes, but they also stop being creative or adaptable, and the food ends up tasting worse.

The paper suggests there is a trade-off: To get a good grade that matches human opinion, the AI needs a little bit of "randomness" (variability). But that same randomness makes the AI inconsistent with itself.

The Human Comparison: Are Humans Any Better?

The researchers also looked at human judges to see if this is an AI-only problem.

Human Experts: When experts graded the same text, they agreed with each other reasonably well, but not perfectly.
Crowd Workers: When regular people (crowd workers) graded the text, they often disagreed with each other and with the experts.
The Surprise: In some cases, the AI judges were actually more consistent with themselves than humans were with each other. However, because the AI's scores were so volatile, it's hard to tell if the AI is actually "right" or just "lucky."

Why Does This Matter? (The "Broken Ruler" Analogy)

Imagine you are trying to measure the height of a tree.

The Old Way: You use a ruler that is slightly bent. It's not perfect, but it's steady.
The New Way (LLM Judge): You use a ruler made of rubber. Sometimes it stretches, sometimes it shrinks. Even if you measure the same tree three times, you get three different numbers.

If you use a rubber ruler, you can't trust the measurement. The paper argues that using an LLM as a judge is currently like using a rubber ruler.

If you run the test once, you get a score.
If you run it again, you get a different score.
If you run it a third time, you get a third score.

Because the score changes every time, we don't know if the AI is actually good at judging quality, or if it's just guessing.

What Did They Find in Different Tasks?

The researchers tested three different types of "games" the AI had to play:

True or False (SummaC): Is the summary factually correct? (Binary choice).
- Result: AI judges were inconsistent, but newer, bigger models were slightly better.
The 1-to-5 Star Review (SummEval): Rate the summary on "Coherence," "Fluency," etc.
- Result: The AI was very inconsistent, especially on "Fluency" (how smooth the text sounds). Interestingly, the AI was sometimes better at judging fluency than humans were at agreeing with each other.
The Battle Royale (MT-Bench): Which of two chatbots gave a better answer?
- Result: This was the hardest task. The AI judges were extremely volatile here, often changing their minds about which chatbot was better.

The Takeaway: How to Fix the Roulette Wheel

The paper offers a few practical tips for people who want to use AI judges:

Don't trust a single run: If you ask an AI to grade something, don't just take the first answer. Ask it to grade it three times and take the average or the majority vote. This smooths out the "roulette" effect and gives a score closer to what a human would say.
Don't turn off the randomness: Even though randomness causes inconsistency, turning it off completely makes the AI worse at matching human opinions. You need a little bit of chaos to get the right answer.
Check your own consistency: Before you trust an AI judge, you should check if it agrees with itself. If it can't agree with itself, it probably can't agree with you, either.

Summary

The paper reveals that AI judges are currently unreliable because they give different answers to the same question every time they are asked. They are like a roulette wheel rather than a steady scale. While newer AI models are slightly more consistent, they still struggle to be reliable. The best solution right now is to ask the AI to play the game multiple times and average the results, rather than trusting a single "spin" of the wheel.

1. Problem Statement

As Natural Language Generation (NLG) becomes ubiquitous, evaluating model outputs has shifted from traditional metrics (e.g., BLEU, ROUGE) to LLM-as-a-Judge frameworks, where Large Language Models (LLMs) assign scores or rankings to generated text. While these frameworks offer scalability and better alignment with human preferences than n-gram metrics, they suffer from a critical, often overlooked flaw: low intra-rater reliability (self-reliability).

The authors identify that LLM judges exhibit significant self-inconsistency. When the same LLM is prompted with identical inputs, hyperparameters, and settings across multiple runs, it frequently produces different ratings. This variance renders individual judgments potentially arbitrary and makes it difficult to determine the true quality of the LLM's evaluation capabilities. Furthermore, existing studies often rely on metrics like correlation or accuracy to compare LLMs against humans, which fail to account for chance agreement, potentially inflating perceived reliability.

2. Methodology

The authors conducted a comprehensive empirical study across three distinct NLG benchmarks to quantify self-reliability and its impact on evaluation performance.

Datasets & Tasks

SummaC: A binary classification task (Consistent vs. Inconsistent) for factual consistency in summarization.
SummEval: A multi-metric summarization evaluation using a 1–5 Likert scale across four dimensions: Coherence, Consistency, Fluency, and Relevance.
MT-Bench: A multi-turn conversation ranking task where a judge selects the better response between two models (Model A, Model B, or Tie).

Experimental Setup

Models: Three LLMs were used as judges: Llama-3.1-70B-Instruct, DeepSeek-R1-Distill-Qwen, and Qwen3-32B.
Procedure: Each judge was run three times on the same dataset with identical prompts and settings to measure intra-rater variance.
Metrics for Agreement:
- Krippendorff's Alpha ( $\alpha$ ): The primary metric used to measure agreement. It was chosen over accuracy or correlation because it corrects for chance agreement and handles missing data, varying numbers of raters, and different data types (nominal, ordinal, interval).
- Distance Functions: Nominal distance for binary tasks (SummaC); Ordinal distance for Likert scales and rankings (SummEval, MT-Bench).
- Balanced Accuracy: Used for SummaC to account for class imbalance.
Comparisons: The study compared:
- Self-Reliability: Agreement of an LLM with itself across runs.
- Inter-Rater Reliability: Agreement between LLMs and Human judges (Experts and Crowdworkers/Turkers).
- Performance Impact: The effect of "Majority Voting" (aggregating multiple runs) vs. "No Sampling" (setting temperature to 0) on agreement with human gold standards.

3. Key Contributions

Quantification of Self-Inconsistency: The paper provides the first systematic quantification of low intra-rater reliability in LLM judges across diverse tasks (summarization, fact-checking, and dialogue).
The Variance-Performance Trade-off: The authors demonstrate a counter-intuitive finding: Disabling sampling (setting temperature to 0) to force deterministic outputs reduces the LLM's agreement with human judgments. This suggests that the inherent stochasticity of LLMs, while causing self-inconsistency, may actually aid in exploring the solution space to find answers closer to human consensus.
Critique of Evaluation Metrics: The study highlights that standard metrics like Accuracy and Correlation overestimate agreement between LLMs and humans because they do not correct for chance. Krippendorff's Alpha reveals significantly lower, and more realistic, agreement levels.
Task-Specific Variability: The inconsistency is not uniform; it varies significantly by task (e.g., lower in MT-Bench than SummaC) and by metric (e.g., "Fluency" is notoriously hard for LLMs to rate consistently).

4. Key Results

Self-Reliability (Intra-Rater)

SummaC: Krippendorff's $\alpha$ ranged from 0.32 (Llama 3.1) to 0.79 (Qwen 3). Even the best model (Qwen 3) barely reached the 0.8 threshold for "good" agreement.
SummEval: Self-reliability varied by metric. While Coherence and Consistency showed moderate reliability, Fluency had very low reliability across all models.
MT-Bench: Self-reliability was the lowest, with $\alpha$ values around 0.26–0.56. Qwen 3 agreed with itself on only 61.3% of cases across three runs.
Conclusion: LLM judges are highly volatile, often more so than human judges.

Agreement with Humans

Metric Bias: Using Accuracy, LLMs appeared to agree with humans at high rates (e.g., 0.827 for humans vs. humans in MT-Bench). However, using Krippendorff's Alpha, human-human agreement dropped to 0.478, and LLM-human agreement dropped even further (e.g., Qwen 3 vs. Humans: 0.426).
Expert vs. Crowd: Human experts showed higher agreement with each other than crowdworkers did. LLMs showed modest agreement with experts but diverged significantly from crowdworkers, suggesting LLMs may align better with expert reasoning but still fall short of the "gold standard."

Impact of Sampling Strategies

Majority Voting: Aggregating results from multiple runs (e.g., taking the majority vote of 3 runs) improved agreement with human labels compared to a single run.
Deterministic Mode (Temp=0): Running the model without sampling (to eliminate variance) degraded performance. For example, on SummaC, Qwen 3's balanced accuracy dropped from 80.6% (Majority Vote) to 79.2% (No Sampling). This indicates that forcing determinism harms the model's ability to align with human preferences.

5. Significance and Recommendations

Significance:
The paper challenges the current paradigm of "single-run" LLM evaluation. It reveals that the "gold standard" of LLM-as-a-Judge is unstable. If an LLM cannot agree with itself, its utility as a reliable evaluator is compromised. The findings suggest that the stochastic nature of LLMs is not just a bug but a feature that aids in reaching human-aligned consensus, and removing it is detrimental.

Recommendations for Future Research:

Report Self-Reliability: Evaluation frameworks must report intra-rater reliability (e.g., Krippendorff's Alpha) alongside inter-rater agreement.
Use Chance-Corrected Metrics: Avoid Accuracy/Correlation for judge evaluation; use Krippendorff's Alpha or Cohen's Kappa.
Aggregate Multiple Runs: Instead of a single inference, use majority voting or aggregation of multiple runs to stabilize scores and improve alignment with human judgment.
Do Not Force Determinism: Avoid setting temperature to zero to fix consistency issues, as this degrades performance.
Measure Human Self-Reliability: Future benchmarks should include repeated human annotations to establish a realistic upper bound for LLM self-reliability.

In conclusion, the authors argue that while LLM-as-a-Judge is a scalable alternative to human evaluation, its current implementation suffers from significant "Rating Roulette" (randomness). Robust evaluation requires acknowledging this variance, using appropriate statistical metrics, and leveraging multiple inference runs rather than relying on single-shot judgments.