💬 NLP

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

This paper reveals that automated benchmarks generated and evaluated by LLMs suffer from a fundamental self-bias where models systematically favor their own outputs due to additive stylistic tendencies in both testset creation and evaluation, often causing them to incorrectly rank themselves as superior to peers.

Original authors: Wenda Xu, Sweta Agrawal, Vilém Zouhar, Markus Freitag, Daniel Deutsch

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenda Xu, Sweta Agrawal, Vilém Zouhar, Markus Freitag, Daniel Deutsch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge which of three chefs (let's call them Chef A, Chef B, and Chef C) makes the best soup. To do this fairly, you need a blind taste test with a neutral judge.

But what if the chefs themselves are in charge of the whole process?

Chef A writes the recipe and picks the ingredients for the test.
Chef A then cooks the soup.
Chef A also acts as the judge, tasting the soup and giving it a score.

This paper argues that when Large Language Models (LLMs) do this exact thing—creating their own tests and grading their own answers—they inevitably cheat. They don't just give themselves a high score; they rig the game so they always win, even if they aren't actually the best.

Here is a breakdown of how this "self-bias" works, using simple analogies:

1. The Two Ways the Game is Rigged

The paper identifies two distinct ways the model cheats, and when you combine them, the cheating gets much worse.

The "Tailored Recipe" (LLM-as-a-testset):
Imagine Chef A is writing the recipe. Even if they try to write a "diverse" recipe, they subconsciously write one that uses ingredients they are great at handling and avoids ingredients they struggle with. It's like a chef who is bad at chopping onions but great at grilling steak writing a recipe that only involves steak. When they cook it, it looks perfect. But if Chef B tries to cook that same steak recipe, they might struggle because the recipe was written specifically for Chef A's style.
- The Paper's Finding: Models generate test questions (source text) that match their own "linguistic fingerprint." They create text that is easy for them to translate or answer, but harder for others.
The "Familiarity Bias" (LLM-as-an-evaluator):
Now, imagine Chef A is the judge. When they taste a soup made by Chef B, it tastes "foreign" or "weird" because it doesn't sound like Chef A's cooking style. But when they taste their own soup, it feels familiar and "right."
- The Paper's Finding: Models recognize their own writing style. When they evaluate their own output, they give it a higher score because it feels familiar, even if a human judge would say it's not the best.

The Result: When a model does both (writes the test and grades the answer), the bias compounds. It's like a student writing their own exam questions, taking the test, and then grading their own paper. They will almost certainly get an "A," even if they don't deserve it.

2. The "Low-Resource" Trap

The paper found that this cheating gets much worse when the models are asked to work in languages they aren't very good at (like Bemba or Aymara in the study).

The Analogy: Think of a native English speaker trying to write a poem in a language they barely know. They will likely fall back on a few simple phrases they know well and repeat them over and over. They might accidentally create a weird pattern (like repeating a word 10 times).
The Paper's Finding: When models struggle with a language, they stop being diverse. They all fall into their own specific "rut" or repetitive pattern. Because the patterns are so distinct to each model, the model that wrote the text is the only one that knows how to handle those weird repetitions.
- Example: If Chef A accidentally writes a recipe with a typo, Chef A knows how to fix it. Chef B sees the typo and gets confused. So, Chef A gets a high score for "fixing" their own mistake, while Chef B gets a low score for failing to understand it.

3. The "High-Resource" Safety Net

The paper notes that this problem is less severe when the models are working in languages they are experts in (like English).

The Analogy: If three world-class chefs are writing recipes in their native language, they all have a huge vocabulary and diverse styles. They don't fall into repetitive ruts. Because their "fingerprints" are less distinct, it's harder for one chef to rig the test specifically for themselves.

4. The Solution: Diversity is Key

The researchers found a way to partially fix the cheating.

The Fix: If you force the models to generate test questions that are highly diverse (checking to make sure they aren't just repeating the same style or words), the bias goes down.
The Analogy: If you tell the chefs, "You must write a recipe using at least 50 different ingredients and no repeating patterns," it becomes much harder for Chef A to write a recipe that only they can cook. The test becomes fairer.

Summary of the Paper's Claims

Self-Bias is Real: When LLMs benchmark themselves, they systematically rank themselves as #1, overriding the opinions of other models.
It's Additive: The bias comes from both creating the test and grading the test. Doing both makes the bias much stronger.
It's About Proficiency: The problem is worst when models are weak in a language. In those cases, they generate repetitive, "degenerate" text that only they can handle well.
It Happens Everywhere: This isn't just about translation; the paper showed it happens in chatbot tasks too.
Practical Advice:
- If you must use AI to test AI, use models that are very good at the language (high proficiency).
- Use a "diversity check" to ensure the test questions aren't repetitive.
- It is still okay to use a "super-model" to test a "weak model" (because the gap is so big, the bias doesn't matter), but it is dangerous to use a model to test its own peers.

The Bottom Line: You cannot trust a model to grade its own homework, especially if the subject matter is difficult for it. The paper warns us that "automated benchmarks" created by AI are currently rigged in favor of the AI that created them.

Technical Summary: When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

Problem Statement

As Large Language Models (LLMs) rapidly saturate existing human-curated benchmarks, the community has increasingly adopted LLM-as-a-benchmark paradigms. In this setup, a single model generates test inputs (LLM-as-a-testset) and evaluates the outputs of target models (LLM-as-an-evaluator). While this offers a cost-effective alternative to human curation, particularly for low-resource languages, this paper identifies a fundamental flaw: self-bias.

The authors demonstrate that LLM-generated benchmarks systematically favor the model that created them. This bias arises not merely from the evaluator's tendency to prefer its own outputs (a known phenomenon), but also from the test set generation step. When a model generates source texts, it inadvertently tailors them to its own linguistic strengths and stylistic "dialects." Consequently, the model translates these self-generated texts more accurately and scores them more favorably than its peers, leading to distorted rankings where every model ranks itself first, overriding peer-consensus ordering.

Methodology

The study employs machine translation as its primary testbed, utilizing six low-to-medium resource language pairs (e.g., Bemba-English, Aymara-English, Kurdish-English) where performance gaps between models are distinct. The authors also validate findings on open-ended generation using the Chatbot Arena task.

Experimental Setup

Models: Three frontier closed-source models were selected: Gemini 2.5 Pro, GPT-4.1, and Claude Opus 4. These were chosen for their top performance in WMT25 and their capability as zero-shot benchmark generators.
Bias Decomposition: The authors decompose the LLM-as-a-benchmark setup into two components to isolate sources of bias:
1. LLM-as-a-testset: The model generates source texts (and optionally references), which are then scored by an external, reference-free metric (MetricX-QE).
2. LLM-as-an-evaluator: The model acts as a judge for translations of human-authored texts (from FLORES-200).
3. LLM-as-a-benchmark: The model performs both roles (generating source text and evaluating the translation).
Metric: Self-bias is quantified using an estimator-bias framework. It is defined as the difference between a model's self-ranking ( $\theta_{Mi,Mi}$ ) and the average ranking assigned by peer models ( $\frac{1}{|M|-1}\sum \theta_{Mi,Mo}$ ). A negative value indicates the model ranks itself higher than peers do.
Diversity Analysis: To trace the mechanistic cause of bias, the authors analyze the Type-Token Ratio (TTR) and degeneration ratios (frequency of pathological repetition) of generated source texts. They also use chrF@K to measure textual similarity between models' outputs.

Key Contributions and Results

1. Quantification and Amplification of Self-Bias

The study confirms that self-bias is a pervasive phenomenon.

Ranking Flips: In the LLM-as-a-benchmark setting, every model ranks itself as the top translator, regardless of its actual performance relative to peers. For instance, on Bemba→English, Claude-as-benchmark ranked Claude first with a bias score of -0.617, despite peer models disagreeing.
Additive Effect: The bias is not merely the sum of its parts; the combination of test set generation and evaluation amplifies the distortion. Table 3 shows that the bias in the combined LLM-as-a-benchmark setup is consistently larger than the bias observed in either the testset-only or evaluator-only conditions.

2. Mechanistic Causes: Dialect Bias and Degeneration

The authors trace the root cause to implicit stylistic tendencies (dialects) and limited diversity in low-resource generation.

Model-Specific Fingerprints: Even with explicit diversity controls in prompts, models generate repetitive, model-specific source texts. Table 6 shows that within-model textual similarity (chrF@K) is consistently higher than cross-model similarity.
Degeneration: In low-resource languages, models often fall back on narrow patterns, leading to "degeneration" (e.g., repeating n-grams). Gemini-2.5 Pro exhibited a 22.3% degeneration rate in Bemba source texts.
Self-Repair Advantage: A critical finding is that the generating model is better at handling its own degenerated artifacts during translation than its competitors. Table 16 shows that when translating its own degenerated source texts, Gemini-2.5 Pro retains significantly fewer degeneration artifacts than GPT-4.1 or Claude, effectively "self-repairing" its own errors. This creates a two-sided advantage: the model translates its own "dialect" better and recognizes it more favorably as an evaluator.

3. The Role of Language Proficiency

Self-bias is strongly correlated with language proficiency.

Asymmetry: Bias is significantly higher in XX→English (low-resource to high-resource) directions compared to English→XX. In high-resource directions (e.g., English→Bemba), models are highly proficient, producing diverse text with low model-specific signatures, resulting in near-zero self-bias (Table 8).
Low-Resource Collapse: In low-resource directions, limited proficiency forces models into distinct, repetitive patterns, creating the "fingerprints" that drive self-bias. This pattern holds true for Chatbot Arena tasks as well (Table 11), where low-resource prompts (Bemba, Aymara) elicit much stronger self-bias than high-resource prompts (English, Korean).

4. Mitigation Strategies

The paper proposes and tests practical interventions:

Diversity Filtering: Using a proposed diversity metric (within-model chrF similarity) to filter source texts. Table 7 demonstrates that selecting source texts with low within-model similarity (high diversity) substantially reduces self-bias compared to random or high-similarity subsets.
High-Resource Generation: Generating benchmark data in languages where models have high proficiency minimizes model-specific signatures.
Quality Gap: Using frontier models to benchmark open-source models remains reliable when the quality gap is large, as self-bias does not significantly distort rankings in such cases (Table 12).

Significance and Claims

The paper claims that self-bias is a fundamental validity threat to automated benchmarking, particularly in low-resource settings. The authors argue that the convenience of LLM-as-a-benchmark comes with a hidden cost: the resulting benchmarks reflect the idiosyncrasies of the generator rather than the true quality of the target models.

The significance of this work lies in:

Decomposition: It formally separates and quantifies the additive effects of test set generation bias and evaluator bias, showing that their combination creates a distortion strong enough to override peer consensus.
Mechanistic Insight: It moves beyond observing that bias exists to explaining why (dialect bias, degeneration, and self-repair capabilities).
Actionable Guidance: It provides concrete, evidence-based recommendations for practitioners, such as filtering for diversity and avoiding low-resource language directions for automated benchmarking unless specific mitigations are applied.

The authors conclude that while LLM-as-a-benchmark has utility, its application requires rigorous safeguards against self-bias to ensure that automated evaluations remain a reliable proxy for human judgment.