Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMs

Technical Summary: Do Political Opinions Transfer Between Western Languages?

Problem Statement
Public opinion surveys consistently demonstrate cross-cultural differences in political stances across socio-linguistic contexts. However, it remains unclear whether these human-level variations translate to cross-lingual differences in Multilingual Large Language Models (MLLMs). While prior research has extensively evaluated political biases in English-only LLMs, the behavior of MLLMs across different Western languages is under-explored. The central research question is whether MLLMs reflect distinct socio-cultural political opinions for each language they support, or if political opinions are transferred uniformly across languages due to the dominance of English in pretraining and alignment data.

The authors define two competing hypotheses:

• H1 (Opinion Transfer): Political opinions transfer between languages; aligning a model in one language affects all supported languages uniformly.
• H2 (Separate Opinions): MLLMs maintain distinct political opinions for each language, allowing for socio-cultural divergence.

Methodology
The study employs a two-phase experimental design involving 15 unaligned instruction-tuned MLLMs of varying sizes (ranging from 1B to 35B parameters) across five Western European languages: English, German, French, Spanish, and Italian.

1. Robustness Filtering (RQ1):

   • Data: The authors utilize ProbVAA, a dataset of 239 statements from European voting advice applications (VAAs), categorized into eight policy issues (e.g., environmental protection, law and order) with known left/right stances.
   • Robustness Check: To ensure model reliability, the authors apply a robustness framework (adapted from Ceron et al., 2024) testing consistency against prompt variations, including paraphrasing, negation, answer inversion, and template wording changes. Only models passing at least 50% of these tests were retained for cross-lingual analysis (six models: Phi3.5-3B, Llama3.1-8B, Aya23-8B, Mixtral8x7B, CommandR-35B, GPT3.5-turbo).
   • Analysis: A beta regression model quantifies the effects of language and model on political stances, with English as the reference language. Non-parametric Kruskal-Wallis tests assess significant differences between languages.

2. Political Alignment and Transfer (RQ2):

   • Alignment Strategy: Two robust models (Phi3.5-3B and Llama3.1-8B) were aligned to exhibit more left-leaning or right-leaning views using Direct Preference Optimization (DPO).
   • Data Source: Alignment datasets were constructed from the Manifesto Project (MARPOR) corpus, specifically filtering for English-language party manifestos. Two annotation approaches were used:
     • RiLe Approach: Using existing right-left scores based on MARPOR codes.
     • Policy Issue Approach: Annotating manifesto sentences specifically for agreement/disagreement with the eight target policy issues.
   • Evaluation: The aligned models were re-evaluated on the ProbVAA dataset in all five languages. Additionally, an open-ended evaluation task was conducted where models generated opinionated summaries on four policy issues, with stances detected by a separate LLM (Llama-3.1-70B-Instruct).

Key Results

• RQ1: Unaligned Models:

  • Robustness: Most languages exhibited similar robustness levels, indicating that political opinions in non-English languages are not merely artifacts of poor generation but are stable.
  • Cross-Lingual Differences: The regression analysis revealed no significant differences in overall political stances between English and other Western languages. While minor, statistically significant differences appeared on specific policy issues (e.g., German, Spanish, and Italian showed slightly stronger support for environmental protection than English), these were not systematic across the board.
  • Model Variance: Significant differences were found between models (e.g., Mixtral8x7B was the most left-leaning), but not systematically between languages within the same model.

• RQ2: Aligned Models:

  • Uniform Shift: When models were aligned using English-only data, the resulting political shifts occurred almost uniformly across all five languages.
  • Validation of H1: The aligned models moved consistently toward the target ideology (left or right) in German, French, Spanish, and Italian, mirroring the shift in English. This confirms that aligning an MLLM in English effectively transfers that alignment to other Western languages.
  • Open-Ended Confirmation: The open-ended evaluation corroborated these findings, showing that the "Left Score" of generated text shifted consistently across languages based on the alignment direction.

Key Contributions

1. Robust Cross-Lingual Evaluation: The paper provides a detailed, robustness-aware evaluation of political opinions in multiple unaligned MLLMs across five Western languages, filtering out models that fail to provide consistent stances.
2. Alignment Transfer Analysis: It offers a thorough analysis of how political alignment in English propagates to other languages, demonstrating that English-centric alignment data induces uniform ideological shifts across the model's entire multilingual capability.
3. Methodological Insight: The study highlights that achieving distinct socio-linguistic or cultural alignment in MLLMs is challenging because the models' internal representations appear to be dominated by English, leading to a "first language" effect where alignment in one language dictates behavior in others.

Significance and Claims
The authors conclude that in Western language contexts, political opinions transfer between languages within MLLMs. This finding challenges the assumption that MLLMs can easily represent distinct socio-cultural political nuances for each language they support.

The significance of this work lies in identifying a fundamental methodological constraint: creating alignment datasets for individual languages in isolation is insufficient because languages are interdependent within MLLMs. Consequently, achieving explicit socio-linguistic, cultural, and political alignment that respects regional differences is inherently difficult. The paper suggests that future work must account for these cross-lingual interaction effects when deploying MLLMs in political tasks across diverse socio-linguistic contexts. The authors remain modest regarding the causal mechanisms, noting that while the transfer is evident, the specific architectural or training dynamics driving it (e.g., English dominance in pretraining vs. post-training alignment) require further investigation.