Language Bias under Conflicting Information in Multilingual LLMs

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed to synthesize information from diverse sources, a task that frequently involves encountering contradictory data. While previous research has established that LLMs struggle to detect and report conflicts (often ignoring contradictions and asserting a single answer), it remains unclear if language acts as a determining factor in which piece of conflicting information is prioritized.

The authors investigate whether multilingual LLMs exhibit systematic biases toward specific languages when presented with conflicting "needles" (facts) embedded within a "haystack" (context). Specifically, they ask:

• Do models fail to detect conflicts in multilingual settings?
• Is there a consistent preference for information presented in certain languages over others?
• Do these biases vary based on the model's geographic origin (e.g., models trained in Mainland China vs. the West)?

2. Methodology

The study extends the "Conflicting Needles in a Haystack" (NIAH) paradigm to a multilingual setting.

• Dataset Construction:

  • Haystacks: Created using authentic news articles (sourced from Creative Commons-licensed outlets like Wikinews and ProPublica). The base text is English, machine-translated into Chinese (cmn), German (deu), Russian (rus), and Turkish (tur).
  • Needles: Two conflicting pseudo-facts (e.g., different names for a band's lead vocalist) are inserted into two different articles within the same haystack.
  • Configuration: For each experiment, a haystack contains $N=34$ articles (~25,000 words total). Two specific articles contain the conflicting needles ($X_1$ and $X_2$). The languages of these two articles are swapped between contrastive pairs (e.g., Pair A: Needle 1 in English/Needle 2 in Chinese; Pair B: Needle 1 in Chinese/Needle 2 in English) while keeping the content identical.
  • Scale: 900 unique haystacks were generated per model, resulting in 4,500 queries per model across 5 prompt languages.

• Models Evaluated:

  • A diverse set of 12 multilingual LLMs ranging from 4B to 27B parameters.
  • Origins: Includes models from the USA (GPT-5 variants, Gemma, Llama, Command-R), Canada, France, and Mainland China (GLM, Qwen, Yi).
  • Sizes: Ranges from small (4B) to large (27B) to test scalability.

• Experimental Setup:

  • Task: The model is prompted to retrieve a specific entity (e.g., "Who is the lead vocalist of Cinderfax?") from the haystack.
  • Metrics: The output is analyzed to see if the model: (1) retrieves both conflicting answers, (2) retrieves only one, (3) retrieves neither, or (4) hallucinates.
  • Bias Detection: By comparing the outcomes of contrastive pairs (where only the language of the needle changes), the authors determine if a model systematically prefers the answer presented in Language A over Language B.
  • Statistical Analysis: A hierarchical Bayesian model (using Stan) was fitted to estimate log-odds bias parameters ($b_{l1,l2}$) across models and prompt languages.

3. Key Contributions

1. Multilingual NIAH Paradigm: Introduction of a controlled experimental framework using manually constructed, naturalistic templates to simulate real-world multilingual information retrieval with conflicting data.
2. Failure to Detect Conflict: Empirical evidence that even state-of-the-art models (including GPT-5.2) overwhelmingly fail to identify contradictions, confidently asserting a single answer in the vast majority of cases, regardless of context length.
3. Systematic Language Bias: Discovery of a consistent, cross-model bias where models prefer information in specific languages. Notably, Russian is consistently disfavored, while Chinese is favored in long-context scenarios.
4. Geographic Origin Analysis: Demonstration that while bias patterns are largely consistent between Western and Chinese-trained models, there are subtle differences (e.g., a slight bias against English in Chinese-trained models).

4. Key Results

A. Handling of Conflicting Information (RQ1)

• Conflict Ignorance: All tested models failed to reliably detect conflicts. In monolingual and multilingual settings, the vast majority of queries resulted in the model selecting only one answer without acknowledging the alternative.
• Context Length: As context length increased (from ~1k to ~25k words), the rate of "None" (failure to retrieve any correct answer) increased for some models, but the tendency to pick a single answer over reporting a conflict remained dominant.

B. Language Bias Presence (RQ2 & RQ3)

• Russian Dispreference: Across almost all models and languages, information presented in Russian was significantly less likely to be retrieved than information in other languages. This bias persisted even though models could accurately extract Russian facts in non-conflicting (monolingual) settings.
• Chinese Preference: In the longest context lengths (25k words), there was a strong, consistent bias in favor of Chinese over other languages (except Turkish in some cases).
• Consistency: These biases were observed across models of different sizes and architectures, suggesting a systemic issue in how multilingual LLMs process conflicting multilingual signals.
• Magnitude: While language bias exists, it is not the sole determinant. In over 90% of cases, other factors (unrelated to language) determined which needle was chosen, as the total counts for language pairs were far below the theoretical maximum.

C. Geographic Origin Influence (RQ4)

• Similarity: The general patterns (anti-Russian, pro-Chinese) were similar for models trained in the West (USA, Canada, France) and the East (PRC).
• Nuance: Eastern (PRC) models showed a slightly stronger bias against Russian and a weak bias against English, which was not observed in Western models.
• Conclusion: The training origin influences the strength of the bias but not the fundamental direction of the preference patterns.

5. Significance and Implications

• Reliability of Multilingual Summarization: The findings suggest that LLMs are currently unreliable for summarizing conflicting information from diverse linguistic sources. They do not act as neutral arbiters but rather as biased filters that may silently discard valid information based on its language.
• Geopolitical Neutrality: The consistency of biases across models trained in different political spheres (West vs. PRC) suggests that these biases may stem from pre-training data distributions or architectural limitations common to the field, rather than specific alignment tuning. However, the specific bias against Russian and preference for Chinese in long contexts warrants further investigation into training data composition.
• Safety and Hallucination: The inability to detect conflicts poses a significant safety risk. If a model confidently asserts a false fact because it "prefers" the language of that fact, it can propagate misinformation.
• Future Directions: The authors highlight the need for larger language samples, more diverse model architectures, and research into internal representations to understand why these biases exist. They also note that increasing Chain-of-Thought (CoT) effort might mitigate these issues, though this was not fully explored in this study due to computational constraints.

In summary, the paper reveals a critical, often overlooked flaw in multilingual LLMs: they do not just struggle with finding information in long contexts; they systematically filter conflicting information based on language, potentially silencing valid data from specific linguistic communities (notably Russian speakers) while amplifying others.