💬 NLP

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

This paper reveals that rubric-based LLM-as-a-judge evaluation inherently exhibits model-specific position bias similar to multiple-choice tasks, where the ordering of score options and criteria significantly influences judgments, but demonstrates that applying random permutations to these orders can effectively mitigate the bias and improve alignment with human annotations.

Original authors: Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, Yoshitaka Ushiku

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno, Yoshitaka Ushiku

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot judge (a Large Language Model, or LLM) whose job is to grade student essays. You give the robot a rubric—a list of scoring options like "1: Terrible," "2: Poor," "3: Okay," "4: Good," and "5: Excellent."

The paper "Am I More Pointwise or Pairwise?" asks a simple but surprising question: Does the robot care about where the options are listed on the page?

Here is the breakdown of their findings, using some everyday analogies.

1. The "Menu" Problem

Usually, we think of these rubrics as a simple checklist. But the researchers discovered that for these AI judges, a rubric acts more like a multiple-choice test.

When the robot reads the list, it doesn't just look at the meaning of the scores; it also looks at their position. Just like a human might subconsciously pick the first option on a menu because it's the first thing they see, or the last option because it's the most memorable, these AI judges have a "position bias."

The Finding: Some robots love the first option (the "First-Biased" judge). Others love the last option (the "Last-Biased" judge).
The Surprise: It's not the same for everyone. One model might always pick "Good" just because it's at the top of the list, while a different model might pick "Excellent" just because it's at the bottom. It's a personality quirk of the specific robot, not a flaw in the rubric itself.

2. The "Line-Up" Bias

The researchers also looked at a scenario where the robot has to grade an essay on multiple things at once (e.g., Coherence, Empathy, and Surprise).

Imagine a lineup of suspects. If you ask the police to identify the culprit, they might focus more on the person standing in the front of the line. Similarly, the paper found that the order of the criteria matters.

If "Coherence" is listed first, the robot might give it a slightly different score than if "Coherence" is listed third.
This happens even if the robot is perfectly capable of understanding the text. The order of the questions shifts the final score.

3. The "Shuffle" Solution

So, how do you fix a robot that is easily distracted by where things are written?

The researchers tried a method called Balanced Permutation. Think of it like shuffling a deck of cards.

Instead of asking the robot to grade the essay once with the list in order (1, 2, 3, 4, 5), they asked it to grade the same essay ten times.
Each time, they shuffled the order of the scores (e.g., 3, 1, 5, 2, 4; then 5, 2, 1, 4, 3, etc.).
By averaging the results, the "position bias" cancels itself out, leaving only the true score.

The Catch: The paper found that you don't need a perfect shuffle. You don't need to do all the math to create a perfectly balanced list.

The Analogy: It's like trying to get a fair average temperature. You don't need to measure it at every single second of the day. If you just take a few random measurements throughout the day, you get a pretty good average.
The Result: Randomly shuffling the list just a few times (about 3 to 5 times) works almost as well as doing the complex, perfect shuffle. It's a cheap and easy way to stop the robot from being distracted by the order.

4. Why This Matters (The "Winner" Changes)

You might think, "Okay, the scores change a little bit, but does it really matter?"

The paper says yes, it matters a lot when you are picking a winner.

Imagine a contest with 4 contestants. If the robot's bias changes the scores slightly, the top-ranked contestant might flip.
The researchers found that simply changing the order of the rubric options caused the "winner" to change in 16% to 39% of the cases.
The Metaphor: It's like a race where the finish line moves slightly depending on which lane you are in. If you are judging a race to decide who gets a scholarship, moving the finish line (changing the rubric order) could accidentally give the prize to the wrong person.

Summary

The Problem: AI judges are secretly influenced by the order of the options they see, just like humans are.
The Quirk: Some AI models love the first option; others love the last.
The Fix: You can fix this by asking the AI to grade the same thing multiple times with the list shuffled in different orders.
The Shortcut: You don't need a perfect shuffle; just a few random shuffles are enough to get a fair result.
The Impact: If you don't fix this, you might accidentally pick the wrong "best" answer, even if the AI is very smart.

The paper concludes that we need to treat these rubric-based AI judges like they are taking a multiple-choice test, where the position of the answer matters, and we need to shuffle the deck to get the truth.

Technical Summary: Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

Problem Statement

While Large Language Models (LLMs) are increasingly used as evaluators (LLM-as-a-Judge), prior research has predominantly focused on biases in point-wise (single response scoring) and pair-wise (comparative selection) paradigms. This paper identifies a previously underexplored source of bias in rubric-based evaluation, a paradigm where models assign scores based on explicit criteria and score ranges.

The authors posit that rubric-based evaluation implicitly functions as a multiple-choice task: given a text, the LLM must select the rubric option that best aligns with the content. Consequently, the ordering of score options (e.g., "1. Worst" to "5. Perfect") or the ordering of multiple criteria within a single prompt may induce position bias. Unlike pair-wise settings where swapping two responses is a known mitigation, rubric-based settings involve multiple options, making simple swaps ineffective. The paper investigates whether LLMs exhibit systematic preferences for score options or criteria based solely on their position in the prompt list, and whether this bias affects downstream rankings and human alignment.

Methodology

The study employs a balanced permutation protocol to disentangle the semantic value of scores/criteria from their positional influence.

Experimental Settings:
- Setting 1 (Score-Option Ordering): The model evaluates a text against a rubric with $n$ score options (e.g., 1–5). The order of these options is permuted.
- Setting 2 (Criterion Ordering): The model evaluates a text against multiple criteria (e.g., Coherence, Empathy). The order of these criteria is permuted.
Balanced Permutation Strategy:
To neutralize position bias, the authors construct balanced orderings where each score or criterion appears an equal number of times at each position across multiple evaluation runs.
- For $n$ options, they generate $2n$ orderings using forward and reverse cyclic rotations (e.g., for 5 scores: [1,2,3,4,5], [2,3,4,5,1], ..., [5,4,3,2,1], [4,3,2,1,5]).
- This ensures that if a model has a positional preference (e.g., favoring the first option), it is averaged out when aggregating results across the balanced set.
Comparison Strategies:
The authors compare three budget-matched strategies:
- BALANCED: The $2n$ cyclic rotations.
- RANDOM: $K$ randomly ordered variants (where $K$ matches the budget of the balanced set).
- FIXED: A single canonical ordering repeated $K$ times.
Datasets and Models:
- Datasets: MT-Bench, Vicuna-Bench, HANNA (story generation), and SummEval (summarization).
- Judges: Six open-weight models across three families (GPT-OSS, Qwen3.5, Gemma-3) with varying sizes (9B to 120B).
- Metrics: Chi-squared ( $\chi^2$ ) goodness-of-fit tests for uniformity, Cramér's $V$ for bias strength, Pearson/Spearman correlation with human ratings, and Kendall $\tau$ for downstream ranking stability.

Key Results

1. Existence and Nature of Position Bias

Statistical Significance: Rubric-based evaluation exhibits statistically significant position bias across all tested models and datasets. The bias is not uniform; it is model-specific.
- Some models (e.g., GPT-OSS-20B) are first-biased (preferring the first option).
- Others (e.g., Gemma-3-27B, Qwen3.5-27B) are last-biased (preferring the final option).
- Some (e.g., GPT-OSS-120B) are nearly uniform.
Orthogonal Bias Axis: A second, independent bias exists regarding the order of criteria in multi-criterion prompts. The position of a criterion significantly affects its assigned score, with some criteria gaining up to 0.80 points depending on their placement.

2. Mitigation and Human Alignment

Permutation vs. Randomization: The study finds that balanced permutation offers no significant advantage over random ordering when the number of aggregated judgments ( $K$ ) is sufficient (e.g., $K=10$ ). The improvement in correlation with human scores comes primarily from variance reduction (aggregating distinct orderings) rather than the mechanical elimination of position bias.
Conditional Improvement: De-biasing (via permutation) only improves human correlation for models that exhibit strong initial bias. For models with low inherent bias, permutation does not yield significant gains.
Budget Efficiency: The benefits of aggregation saturate early. For models that benefit, roughly two-thirds of the improvement is achieved with just $K=3$ random orderings, and 85% by $K=5$ .

3. Downstream Impact

Rank Reversal: Even if average score correlations with humans remain similar, position bias significantly impacts ranking. Changing the rubric order alone caused the top-1 candidate to flip in 16% to 39% of prompts across different judges.
Implications: This variance is critical for applications like "Best-of-N" selection, Reinforcement Learning from AI Feedback (RLAIF), and leaderboard rankings, where the specific candidate selected matters more than the average score.

Significance and Claims

The paper reframes rubric-based LLM-as-a-Judge as an implicit multiple-choice problem subject to measurable, model-specific position bias.

Practical Guidance: The authors argue that while "removing the bias" mechanically is possible via balanced permutations, the practical goal of improving human correlation is better served by simple randomization of the rubric order. A small number of random permutations ( $K \approx 3$ to $5$) is sufficient to mitigate the error introduced by this bias for the majority of models.
Scope of Impact: The bias is most detrimental in selection scenarios (ranking candidates) rather than absolute scoring. The authors caution that relying on a single canonical rubric order can lead to unstable downstream decisions, even if the model appears unbiased on average.
Limitations: The study is limited to open-weight models and does not extend to the most recent closed-source models or direct applications in model training pipelines due to budget constraints.

In conclusion, the paper demonstrates that rubric ordering is a non-trivial source of noise in LLM evaluation. While complex balancing protocols are theoretically sound, a low-cost strategy of aggregating a few randomly ordered variants is statistically indistinguishable from exact balancing and effectively reduces the risk of rank reversal in practical applications.