💬 NLP

Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG

This paper reveals that multilingual Retrieval-Augmented Generation models exhibit a "linguistic nepotism" bias where they preferentially cite English sources over more relevant documents in other languages, particularly for lower-resource languages and mid-context positions, thereby trading off factual relevance for language preference.

Original authors: Dayeon Ki, Marine Carpuat, Paul McNamee, Daniel Khashabi, Eugene Yang, Dawn Lawrie, Kevin Duh

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dayeon Ki, Marine Carpuat, Paul McNamee, Daniel Khashabi, Eugene Yang, Dawn Lawrie, Kevin Duh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Language Cousin" Problem

Imagine you ask a very smart, well-read librarian (the AI) to write a report on how a specific machine works. You give the librarian a stack of 10 different instruction manuals. Some are in English, some in French, some in Swahili, and some in Korean. All of them are actually correct and helpful.

The paper discovers that this librarian has a secret bias: They love their English cousins.

Even if the French or Korean manual explains the machine perfectly, the librarian is much more likely to quote the English one. If the English manual is actually wrong or irrelevant, but the Korean one is perfect, the librarian will often still choose to quote the English one. They are trading quality (the right answer) for language preference (the familiar language).

The authors call this "Linguistic Nepotism"—favoring a family member (English) over a stranger (other languages), even when the stranger is more qualified for the job.

How They Caught the Librarian in the Act

To prove this wasn't just a coincidence, the researchers set up a very strict, controlled experiment. Think of it like a magic trick where they swap the cards without the magician noticing.

The Setup: They took a perfect English report and translated the source documents into eight different languages (like French, Arabic, Swahili, etc.).
The Control: They kept everything else exactly the same. The content was identical; only the language changed.
The Test: They asked the AI to write a report and cite its sources. They watched closely to see which "language cousin" the AI chose to quote.

They found that the AI's "citation accuracy" (how often it picked the right source) dropped significantly when the source was in a non-English language.

Key Findings: When the Bias Gets Worse

1. The "Middle Child" Effect

The researchers found that where the document sits in the stack matters.

The Analogy: Imagine reading a long book. You remember the beginning and the end clearly, but the middle part gets fuzzy.
The Result: The AI's bias toward English gets strongest when the document is in the middle of the context. If a Swahili document is in the middle of the stack, the AI is very likely to ignore it and grab an English one instead, even if the English one is further away or less relevant.

2. The "Underdog" Penalty

The bias is not the same for all languages.

The Analogy: It's like a teacher who is slightly biased against students who speak a rare dialect, but barely notices the bias against students who speak a common foreign language like French.
The Result: The AI's preference for English is much stronger for "lower-resource" languages (like Swahili or Bengali) than for "higher-resource" languages (like French or Spanish). The AI is most likely to ignore the "underdog" languages entirely.

3. The "Query Language" Mirror

The researchers also tested what happens if you ask the question in a different language (e.g., asking in French).

The Analogy: If you speak to the librarian in French, they suddenly become more friendly to French books.
The Result: The AI tends to prefer citing documents in the same language as the question. If you ask in French, it prefers French sources. However, if the question is in English, it overwhelmingly prefers English sources, regardless of what the other documents say.

The Most Shocking Discovery: Relevance vs. Language

This is the most critical part of the paper. The researchers set up a trap:

Document A: Perfectly relevant to the question, but written in Swahili.
Document B: Completely irrelevant (it talks about something else), but written in English.

The Result: The AI often chose to cite Document B (the irrelevant English one) instead of Document A (the relevant Swahili one).

This proves that for these models, language preference can be stronger than truth. The AI would rather lie and quote a familiar language than tell the truth using an unfamiliar one.

How the AI Makes the Decision (The "Brain Scan")

The authors looked inside the AI's "brain" (its internal layers) to see when this decision happens.

The Analogy: Imagine a committee of judges deciding a winner. In the early rounds, they are confused. But around the 20th round, they suddenly pick a winner and stick with it.
The Result: The AI makes its choice very early in its processing. Once it decides to cite an English document, it rarely changes its mind, even if it sees the correct information in another language later in the process. It "locks in" its bias early.

Summary

The paper concludes that Multilingual RAG systems (AI that reads and writes in many languages) suffer from Linguistic Nepotism. They systematically favor English (and the language of the question) over other languages.

They ignore relevant non-English documents.
They prefer irrelevant English documents.
This bias gets worse for less common languages and for documents buried in the middle of long texts.

The authors warn that if we don't fix this, these AI systems will continue to hide information from non-English speakers and reinforce the idea that English is the only "important" language for knowledge.

Technical Summary: Linguistic Nepotism in Multilingual RAG

Problem Statement

Multilingual Retrieval-Augmented Generation (mRAG) systems are designed to answer knowledge-intensive queries with citation-supported responses across diverse languages. However, a critical open question remains: does the mixture of document languages in the retrieval context inadvertently bias the model's generation and citation behavior?

Existing methods for measuring language preference suffer from two primary limitations:

Lack of Correctness Control: Short-form mRAG studies often estimate preference via information overlap or embedding similarity, which do not directly verify if the cited source actually supports the generated claim.
Confounding Factors: Long-form mRAG studies that measure citation frequency against document distribution are confounded by the varying relevance and informativeness of sources. Furthermore, inline citations are prone to hallucinations, making it difficult to distinguish between true attribution and spurious citations.

Consequently, it remains unclear whether models cite specific languages due to genuine informativeness or a systematic bias that may lead to "linguistic nepotism"—trading off document relevance for language preference.

Methodology

To isolate language preference from other variables, the authors propose a controlled methodology using model internals, illustrated in Figure 1 of the paper. The approach involves four steps:

Synthetic Data Generation (Steps 1–2):
- Translation: Starting with English queries and their relevant evidence documents ( $D_{en}$ ), the authors use Machine Translation (Google Translate API) to create parallel document sets in eight target languages ( $D_{fr}, D_{ar}, D_{ko}, \dots$ ).
- Reference Generation: A strong LLM (OpenAI o3) generates a reference citation-supported report for each query using the English documents. This report is segmented into sentence-level statements, each paired with a specific citation ID.
Verification Filtering (Step 3):
- To ensure evaluation correctness, the authors apply a two-stage filter to the statement-citation pairs:
  - LLM-as-Judge: Multiple judge models (OpenAI o4 mini, QWEN-3 32B, Gemini 2.5 Pro) rank the relevance of documents to each statement. Only statements where the majority of judges agree on the cited document are retained.
  - NLI Entailment: A Natural Language Inference classifier verifies that the cited document logically entails the statement.
- This process yields a verified pool of 792 statements where the citation correctness is guaranteed.
Measurement via Next Token Prediction (Step 4):
- The core metric is the accuracy of predicting the correct citation ID token as the next token.
- Contrastive Contexts: For each verified statement, the model is prompted with the statement and a context where only the specific document to be cited ( $d_{ci}$ ) is presented in a target language $\ell$ , while all other evidence documents remain in English.
- Metric: Citation accuracy ( $Acc(\ell)$ ) is calculated as the frequency with which the model correctly predicts the citation ID when the source is in language $\ell$ .
- Preference Gap: The language preference is quantified as $\Delta(\ell_{target}) = Acc(\ell_{target}) - Acc(en)$ . A negative value indicates a preference for English over the target language.

Experimental Setup

Datasets: The primary dataset is the ELI5 (Explain Like I'm Five) long-form QA dataset (270 queries from WebGPT test set). For relevance vs. language experiments, the MIRACL dataset is used to introduce irrelevant distractors.
Languages: Eight target languages representing diverse resource levels: Arabic, Bengali, French, Korean, Russian, Spanish, Swahili, and Chinese.
Models: Six open-weight LLMs of varying sizes and architectures: LLAMA-3.1 8B, LLAMA-3.3 70B, QWEN-3 8B/14B, GEMMA-3 27B, and AYA23 8B.

Key Results

1. Strong English Preference

Across all six models and eight languages, models exhibit a pronounced tendency to cite English documents when the query is in English.

Magnitude: The citation accuracy drops significantly when the cited document is in a non-English language compared to English.
Resource Correlation: The bias is amplified for lower-resource languages. For instance, Swahili and Bengali show the largest accuracy drops (up to -37.6% in AYA23 8B), whereas higher-resource languages like French and Spanish show smaller decreases.
Position Bias: The "lost in the middle" phenomenon exacerbates this bias. Citation accuracy is lowest when the document appears in the middle of the context, and the gap between English and target language accuracy is widest in this position.

2. Query Language Preference

When the query itself is translated into a target language, models shift their preference toward citing documents in the query language, even if that language is not English.

Models perform best when the cited document matches the query language, even if other context documents are in English.
This suggests a "query-language preference" where the model favors linguistic alignment between the query and the evidence, mirroring human "own-language preference" in scientific citation.

3. Language Outweighs Relevance

In experiments introducing irrelevant distractors, the authors find that models frequently trade off document relevance for language preference.

Irrelevant English vs. Relevant Non-English: When a relevant document is in a target language and an irrelevant document is in English, citation accuracy drops significantly below the baseline. The model is misled by the irrelevant English source.
Relevant English vs. Irrelevant Non-English: Conversely, when the relevant document is in English and the distractor is in a target language, accuracy rises above the baseline. The model more easily dismisses irrelevant non-English distractors.
This indicates that in long-form mRAG, language preference can be a stronger driver of citation behavior than factual relevance.

4. Model Internals Analysis

Using Logit Lens visualization on LLAMA-3.1 8B, the authors observe that:

Models do not initially favor English and then correct themselves later.
Instead, there is a specific transition region (layers 18–22) where citation predictions crystallize. Once the model commits to a citation (correct or incorrect), it largely preserves this choice in subsequent layers.
The gap between correct and incorrect predictions narrows for lower-resource languages, confirming stronger English bias in these cases.

Significance and Claims

The paper claims to provide the first controlled measurement of language preference in long-form mRAG using model internals to hold relevance constant. The significance of these findings lies in:

Revealing Systematic Bias: The study demonstrates that citation choices are not driven solely by informativeness. Models exhibit "linguistic nepotism," systematically favoring English (or the query language) even at the cost of relevance.
Impact on Trust and Equity: These biases risk marginalizing non-English sources, reinforcing existing language hierarchies, and reducing trust in AI-assisted tools for multilingual users, particularly in high-stakes domains like healthcare and journalism.
Methodological Contribution: The proposed framework offers a robust way to evaluate citation behavior by isolating language effects from content relevance, addressing the limitations of prior work that relied on coarse frequency counts or unverified overlap metrics.

The authors conclude that building more robust and inclusive mRAG systems requires acknowledging that language itself exerts a stronger influence than document relevance in current models, necessitating evaluation practices that are transparent and language-aware.