💬 NLP

Exploring the Effects of Alignment on Numerical Bias in Large Language Models

This study identifies alignment as the primary cause of numerical bias in LLM-as-a-judge systems and demonstrates that adjusting the score range is the most effective heuristic strategy to mitigate this bias and improve evaluation performance.

Original authors: Ayako Sato, Hwichan Kim, Zhousi Chen, Masato Mita, Mamoru Komachi

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ayako Sato, Hwichan Kim, Zhousi Chen, Masato Mita, Mamoru Komachi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a team of expert judges (Large Language Models, or LLMs) to grade essays, translate stories, or fix grammar mistakes. You ask them to give a score from 0 to 100. This is the "LLM-as-a-judge" method.

However, the researchers in this paper discovered a weird glitch: these digital judges have a bad habit of being too consistent. Instead of giving a wide variety of scores (like 42, 67, 89, 12), they keep hitting the same number over and over again, like a broken record stuck on "80."

The paper calls this "Numerical Bias." It's like a restaurant critic who, no matter what dish you serve them, always writes "8/10" on their review card. Even if the food is terrible or amazing, they just can't seem to break the pattern.

The Culprit: "Alignment"

The researchers wanted to know: Why does this happen?

They suspected the problem was caused by a process called "Alignment."

Before Alignment: Think of an LLM as a wild, creative artist. It might give you a score of 3, then 95, then 42. It's unpredictable and diverse, but sometimes it doesn't follow your instructions well.
After Alignment: To make the AI more helpful and polite, humans "train" it to follow instructions better. This is like putting the artist in a strict art school. Now, the AI follows rules perfectly, but it becomes a bit robotic. It loses its "wild" diversity and starts clustering its answers around a safe, specific number.

The Discovery: The study compared the "wild" (pre-alignment) models with the "trained" (post-alignment) models. They found that the trained models were much more likely to get stuck on specific numbers (like 8 or 9 out of 10), regardless of whether the input was good or bad. This "stuck" behavior actually made their grading less accurate.

The Good News and The Bad News

The Bad News: This "stuck" behavior (bias) makes the AI a worse judge. If it can't distinguish between a great translation and a bad one because it keeps giving them both an "8," the system fails.
The Good News: Even with this flaw, the "trained" (aligned) models are still better at following instructions than the "wild" ones. They are still the best choice for the job, but they need a little help to fix their grading habits.

How to Fix the Broken Record

The researchers tested three ways to stop the AI from getting stuck on one number:

Turning up the "Temperature" (Randomness):
Imagine the AI is a dart player. "Temperature" is how shaky their hand is. If they are too steady (low temperature), they hit the exact same spot every time. If you make their hand a little shaky (higher temperature), they might hit different spots.
- Result: This helped a little, but if you made them too shaky, their scores became random garbage. It wasn't a perfect fix.
Calibration (Re-calibrating the Scale):
This is like telling the AI, "Hey, you're giving too many 8s. Let's mathematically adjust your brain so you give more 5s and 9s."
- Result: This reduced the bias, but it didn't always make the grading more accurate. Sometimes it just made the AI confused.
Changing the Score Range (The Magic Fix):
This was the most effective solution. Imagine you ask the AI to grade on a scale of 1 to 5. It might get stuck on "4." But if you tell it, "Grade on a scale of 1 to 100," it suddenly has more room to breathe. It stops feeling forced to pick the "safe" middle number and starts using the whole range.
- Result: By simply changing the instructions to allow for a wider range of numbers (e.g., 1–100 instead of 1–5), the AI stopped getting stuck. It gave more varied scores, and its accuracy improved significantly.

The Takeaway

The paper concludes that when we use AI to judge things, we shouldn't just assume the settings are perfect.

Alignment (training the AI to be helpful) accidentally makes it too predictable and repetitive in its scoring.
The Solution: Don't just accept the default settings. If you ask an AI to grade something, try telling it to use a wider range of numbers. It's like telling a nervous student, "You can get any grade from A to F," instead of "You can only get a B." This simple change helps the AI show its true judgment rather than just repeating a safe number.

In short: AI judges are great, but they get "stuck" on specific numbers because we trained them too well. Giving them a bigger playground (a wider score range) helps them break out of that habit and do a better job.

Technical Summary: Exploring the Effects of Alignment on Numerical Bias in Large Language Models

Problem Statement

The "LLM-as-a-judge" paradigm, which utilizes Large Language Models (LLMs) as evaluators for text quality, has emerged as a scalable alternative to traditional metrics like BLEU and ROUGE. However, this approach suffers from numerical bias, a phenomenon where evaluators disproportionately generate specific numeric tokens, leading to skewed score distributions. This bias reduces the distinguishability of evaluations and raises concerns regarding the robustness of the framework, as high correlation with human judgments may be achieved artificially through score clustering rather than genuine assessment capability.

While evaluator LLMs are typically post-alignment models (tuned via instruction and preference tuning) to improve instruction-following, prior research suggests that alignment can reduce output diversity. This study hypothesizes that numerical bias is a side effect of the alignment process, where models become over-consistent in their scoring, failing to capture the nuances of input quality.

Methodology

The study investigates the relationship between model alignment and numerical bias through a comparative analysis of pre-alignment and post-alignment models across three regression tasks:

MTQE (Machine Translation Quality Estimation): Evaluating translation quality without reference texts.
GECQE (Grammatical Error Correction Quality Estimation): Assessing corrected sentences.
LCP (Lexical Complexity Prediction): Predicting word complexity.

Experimental Design

Models: The authors compared four model series (Gemma, Mistral, Llama 3, Qwen 2) in both their base (pre-alignment) and instruct (post-alignment) variants.
Metrics:
- Kurtosis: Used as the primary metric to quantify numerical bias. Higher kurtosis indicates a sharper, more concentrated distribution around specific values.
- Pearson Correlation Coefficient ( $r$ ): Measured to evaluate the alignment between model scores and human gold scores.
Mitigation Strategies (RQ2): The study tested three approaches to mitigate bias in post-alignment models:
1. Temperature Scaling: Varying the temperature parameter to increase output diversity.
2. Distribution Calibration: Applying a generative calibration method (Jiang et al., 2023) to correct the label marginal distribution.
3. Score Range Adjustment: Modifying the specified output range in the prompt (e.g., 1–5, 0–9, 1–100).

Key Results

1. Alignment Induces Numerical Bias

Distribution Shift: Pre-alignment models exhibited varied score distributions with lower kurtosis, likely due to weaker instruction-following leading to random outputs. In contrast, post-alignment models generated scores heavily concentrated around specific values (e.g., 8), resulting in significantly higher kurtosis.
Correlation with Accuracy: A strong negative correlation ($-0.60$) was observed between kurtosis and evaluation accuracy ( $r$ ). Models with higher numerical bias consistently demonstrated lower correlation with human judgments.
Language and Model Variance: The bias was more pronounced in high-resource languages (e.g., English-German, English-Chinese) compared to low-resource languages (e.g., Nepali-English, Sinhala-English). Among models, Gemma exhibited the strongest bias, while Llama and Qwen showed relatively lower kurtosis.

2. Mitigation Effectiveness

Temperature Scaling: Increasing temperature reduced kurtosis (bias) but did not consistently improve evaluation accuracy. In some cases, excessively high temperatures degraded performance.
Distribution Calibration: This method successfully reduced kurtosis in several models but yielded mixed results on accuracy, sometimes degrading performance in models that were already accurate.
Score Range Adjustment: This proved to be the most effective strategy. Adjusting the score range (e.g., expanding to 1–100 or narrowing to 1–5) significantly reduced kurtosis and, in many cases, improved the correlation with human judgments. For instance, in the MTQE task with Gemma, a 1–100 range outperformed both the default range and temperature-tuned results.

Key Contributions

Causal Link Identification: The study provides empirical evidence that numerical bias in LLM evaluators is a direct consequence of the alignment process, distinguishing it from general output diversity issues.
Bias-Accuracy Trade-off: The authors demonstrate that while alignment improves instruction-following, the resulting numerical bias can undermine evaluation accuracy. They propose that kurtosis can serve as a proxy metric for selecting robust evaluators when gold data is unavailable.
Optimization of Score Range: The paper challenges the treatment of score ranges as fixed design choices, demonstrating that they should be treated as tunable hyperparameters. Optimizing the score range is shown to be a more effective heuristic for mitigating bias and enhancing performance than temperature scaling or calibration alone.

Significance and Limitations

The paper concludes that while post-alignment models remain superior to pre-alignment models for evaluation tasks due to better instruction-following, their inherent numerical bias limits their reliability. The findings highlight the need for optimal score range selection as a critical, yet often overlooked, component of LLM-as-a-judge systems.

The authors maintain a modest stance, acknowledging that their mitigation strategies (particularly score range adjustment) are heuristic and task-specific rather than fundamental solutions. They note limitations including the inability to analyze internal alignment data of public models, the restriction to numerical score outputs, and the unresolved impact of alignment on natural language or ranking-based evaluations. Future work is directed toward developing more robust, generalizable strategies to address the root causes of alignment-induced bias.