💬 NLP

Evaluating Scoring Bias in LLM-as-a-Judge

This paper addresses the under-investigated issue of scoring bias in LLM-as-a-Judge systems by defining and quantifying three novel bias types—rubric order, score ID, and reference answer score biases—and proposing a comprehensive framework to mitigate them through improved prompt design and automated evaluation.

Original authors: Qingquan Li, Shaoyu Dou, Kailai Shao, Chao Chen, Haixiang Hu

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qingquan Li, Shaoyu Dou, Kailai Shao, Chao Chen, Haixiang Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a very smart, automated robot teacher to grade essays. This robot is supposed to be fair, objective, and consistent. You give it a student's essay and a rubric (a checklist of what makes a good essay), and it gives the essay a score from 1 to 5.

This paper is about discovering that even the smartest robot teachers are actually quite easily tricked. They don't just look at the essay; they get confused by how the instructions are written, even if the instructions say the exact same thing.

Here is the breakdown of the paper's findings using simple analogies:

The Core Problem: The "Magic 8-Ball" Effect

The researchers found that if you change the formatting of the grading instructions slightly—without changing the actual meaning—the robot's score for the same essay can change drastically. It's like asking a friend, "How good is this movie?" and getting a "5 out of 5" if you ask politely, but a "2 out of 5" if you ask while standing on your head. The movie didn't change, but the answer did.

The paper focuses on Scoring Bias, which is when the robot gives a different absolute score (like a 3 or a 4) just because the prompt was tweaked, not because the answer changed.

The Three Ways the Robot Gets Confused

The researchers identified three specific "tricks" that mess up the robot's grading:

The Menu Order Bias (Rubric Order):
Imagine a restaurant menu. Usually, appetizers are listed first, then main courses. If you flip the menu so the main courses are listed first, does the food taste different? No. But the robot teacher thinks so.
- The Finding: If the grading rules are listed from "1 to 5" (low to high), the robot grades differently than if they are listed "5 to 1" (high to low) or in a random order. The robot gets confused by the sequence.
The Name Tag Bias (Score ID):
Usually, we use numbers like 1, 2, 3, 4, 5. But what if we used letters (A, B, C, D, E) or Roman numerals (I, II, III, IV, V)?
- The Finding: The robot's brain reacts differently to these different "names" for the scores. Sometimes using Roman numerals makes the robot a better grader; sometimes it makes it worse. It's like the robot has a favorite alphabet.
The "Perfect Example" Bias (Reference Answer Score):
Sometimes, teachers give students a "perfect example" of an essay to show what a "5" looks like. The researchers tested what happens if they attach a score to that example.
- The Finding: If you show the robot a perfect example and label it "Score 5," the robot becomes very consistent and accurate. However, if you label that same perfect example as a "Score 3," the robot gets confused and starts giving lower scores to everything else, thinking, "Oh, if the perfect example is only a 3, then this student's essay must be terrible."

The Experiments: Testing the Robots

The researchers tested this on several different "robot teachers" (AI models), ranging from the most powerful ones (like GPT-4o) to smaller, cheaper ones.

The Big Robots vs. The Small Robots: The most powerful robots were less easily confused. They were like a senior teacher who knows the material so well that flipping the menu order doesn't throw them off. The smaller robots were like nervous student teachers; a tiny change in the instructions made their scores jump around wildly.
The Surprise: Sometimes, the "tricks" actually helped! For some robots, using Roman numerals or flipping the order of the rules made them grade more accurately than the standard way. It turns out that the "standard" way humans usually write prompts isn't always the best way for robots.

The Solutions: How to Fix the Grading

Based on their experiments, the authors suggest three ways to make these robot judges more reliable:

Hire the Big Robots: If you need a critical grade (like for a job or a school), use the most powerful AI model available. They are naturally more stable and less likely to be swayed by formatting tricks.
Break the Rules (Intentionally): Don't just copy the standard "1 to 5" list. Try listing the rules from "5 down to 1," or use letters instead of numbers. The paper suggests that doing something slightly "unconventional" but clear can actually stop the robot from falling into its usual bias traps.
Show the Perfect Example (with a 5): If you are going to show the robot a "gold standard" answer to help it grade, make sure you label it as a "5" (the highest score). This anchors the robot's thinking and makes it much more accurate.

The Bottom Line

The paper concludes that "LLM-as-a-Judge" (using AI to grade AI) is not as objective as we thought. The robots are sensitive to the presentation of the rules, not just the rules themselves. To get fair grades, we need to be very careful about how we write the instructions, perhaps using the most powerful models and designing our prompts in ways that are slightly different from what we are used to.

Technical Summary: Evaluating Scoring Bias in LLM-as-a-Judge

Problem Definition

The "LLM-as-a-Judge" paradigm utilizes Large Language Models (LLMs) as automated evaluators to provide scalable feedback for complex tasks. While existing research has extensively investigated biases in comparative evaluations (e.g., determining which of two responses is better), scoring-based evaluations—where an LLM assigns an absolute score to a single response—remain under-investigated. Scoring-based methods are often more practical for industrial applications and standardized quantitative analysis.

This paper identifies a critical gap: scoring-based LLM judges exhibit significant instability when their prompts are perturbed, even when the evaluation target (the response being scored) remains unchanged. The authors define scoring bias as the measurable shift in scores exhibited by an LLM judge when subjected to perturbations in the scoring prompt itself, rather than changes to the evaluation target.

Methodology

Evaluation Framework

The authors propose a comprehensive framework to quantify scoring bias, inspired by prior work on prompt bias. The framework involves:

Prompt Perturbation: Systematically modifying non-target components of the scoring prompt while keeping the evaluation target instance ( $I$ ) constant.
Metrics: Three dimensions of evaluation are used:
- Stability: Measured by Flip Rate (FP) (the percentage of samples where the score changes) and Mean Absolute Deviation (MAD) between perturbed scores and a baseline.
- Accuracy: Measured by Spearman's ( $\rho$ ) and Pearson's ( $r$ ) correlation coefficients between the judge's scores and "golden scores" (derived from advanced LLMs like GPT-4.1 and human annotators).
- Scoring Tendency: Analyzed via score distribution histograms to detect inherent preferences for specific score categories.

Perturbation Types

The study introduces and defines three novel types of scoring bias:

Rubric Order Bias: Altering the sequence of score descriptions in the rubric (e.g., Ascending 1–5, Descending 5–1, or Random order).
Score ID Bias: Changing the identifiers used for scores (e.g., Arabic numerals {1–5}, Letter Grades {A–E}, or Roman Numerals {i–v}).
Reference Answer Score Bias: Investigating the impact of attaching specific scores (1–5) to reference answers provided in the prompt, a factor previously unexplored in the context of robustness.

Datasets and Models

Datasets: The evaluation utilizes four established benchmarks: BiGGen Bench, FLASK, MT Bench, and Vicuna Bench. For datasets lacking reference answers for all score levels (BiGGen Bench and FLASK), the authors developed an automatic data synthesis pipeline (generation-review loop) to create reference answers for scores 1 through 4.
Judge Models: Experiments were conducted on a diverse set of models, including GPT-4o, DeepSeek-V3-671B, Qwen3-32B, Qwen3-8B, and Mistral-Small-24B-Instruct.
Golden Scores: GPT-4.1 was used to generate golden scores, validated to have the highest correlation with human annotations among the tested models.

Key Results

1. Ubiquity of Scoring Bias

Even the most advanced LLMs suffer from substantial scoring biases. Perturbations in prompt structure lead to significant inconsistencies:

Stability: Under perturbations, Flip Rates (FR) ranged from ~16% to over 46% depending on the model and perturbation type. For instance, Qwen3-8B exhibited a 46.22% FR on BiGGen Bench under a simple descending rubric order perturbation, meaning nearly half of its scores changed.
Model Sensitivity: Larger, more capable models (e.g., GPT-4o, DeepSeek-V3) generally demonstrated higher robustness (lower FR and MAD) compared to smaller models (e.g., Qwen3-8B).

2. Non-Linear Impact on Accuracy

Contrary to the assumption that bias is always detrimental, the study found that certain perturbations can improve scoring accuracy relative to golden scores:

Score ID: Using Roman numerals improved the accuracy of GPT-4o, while Letter Grades improved DeepSeek-V3-671B. Conversely, these changes often degraded the accuracy of other models.
Rubric Order: Descending rubric orders improved accuracy for DeepSeek-V3 and Mistral-Small, whereas random orders generally harmed accuracy across all models.

3. Scoring Tendencies and Reference Answers

Inherent Preferences: Different models exhibit distinct scoring distributions (e.g., DeepSeek-V3-671B tended to assign high scores, while Mistral-Small favored score 4).
Reference Answer Impact: The inclusion of a reference answer with a full mark (Score 5) consistently yielded the highest scoring accuracy and stabilized scoring tendencies across models. In contrast, reference answers with lower scores (1–4) often caused models to "pull" their scores toward the reference, reducing accuracy.

Key Contributions

Identification of Novel Biases: The paper formally defines and investigates three previously unstudied scoring biases: rubric order bias, score ID bias, and reference answer score bias.
Evaluation Framework: It proposes the first dedicated framework for quantifying scoring bias, introducing a suite of metrics (stability, accuracy, tendency) and an automatic data synthesis pipeline to create a tailored evaluation corpus.
Empirical Evidence: The study provides empirical evidence that even state-of-the-art LLMs are susceptible to these biases, challenging the reliability of current automated scoring practices.

Significance and Recommendations

The paper argues that the reliability of LLM-as-a-Judge in industrial applications is compromised by these prompt-induced biases. The authors offer actionable insights for designing more robust scoring prompts:

Model Selection: Prioritize larger, more capable models for high-stakes evaluations to mitigate bias.
Prompt Design: Adopt "intentional yet meaningful deviation from convention." This includes using descending rubric orders or non-numeric identifiers (like letter grades) where appropriate, while avoiding chaotic randomization.
Reference Answers: When using reference answers, a full-mark (Score 5) reference is recommended to maximize accuracy and stability, aligning with the logic of metrics like BLEU and ROUGE.

The work concludes that addressing scoring bias is essential for the fairness and reliability of automated LLM evaluation, providing a foundation for future research into debiasing strategies and robust prompt engineering.