Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset

This paper introduces a novel 21-way multiparallel EuroParl dataset to assess political bias in multilingual Large Language Models by demonstrating that translation quality systematically favors majority parties over outsider groups, offering a fairness-based alternative to traditional English survey methods.

The Gist

Imagine you have a giant, super-smart robot translator that speaks 21 different European languages. You ask it to translate a speech from a politician, and it does a great job. But what if the robot secretly has a favorite team? What if it translates speeches from "Team Left" or "Team Right" with extra care and accuracy, while translating speeches from "Team Outsider" with more mistakes and clumsiness?

That is exactly what this paper investigates. The authors, Paul Lerner and François Yvon, wanted to see if modern AI translators are fair to all political groups, or if they have a hidden bias.

Here is the breakdown of their study, explained with some everyday analogies.

1. The Problem: The "English-Only" Survey

Usually, when people check if AI is biased, they ask the AI a bunch of questions in English, like a multiple-choice test. "Do you agree that X is bad?"

• The Flaw: This is like judging a chef only by how they cook a burger. It doesn't tell you if they can cook sushi, pasta, or curry. Also, the questions themselves might be tricky or confusing.

2. The Solution: The "21-Lane Highway"

Instead of asking questions, the authors looked at real speeches from the European Parliament.

• The Dataset: They built a massive new dataset called 21-EuroParl. Imagine a highway with 21 lanes. In the middle lane, there is a speech in German. In the other 20 lanes, there are perfect translations of that exact same speech in 20 other languages (French, Polish, Spanish, etc.).
• The Metadata: Crucially, they tagged every speech with the speaker's political party. It's like knowing exactly which team the player is on before the game starts.
• The Scale: This isn't a small test. It's 1.5 million sentences, covering 1,000+ speakers from 27 countries. It's a massive library of political debate.

3. The Experiment: The "Fairness Race"

The researchers took this massive library and fed it into several popular AI models (like Llama, Qwen, and Gemma). They asked the AI to translate the speeches.

Then, they used a clever scoring system called the Borda Count.

• The Analogy: Imagine a race with 8 runners (the 8 major political parties). In every single translation task (e.g., German to French), the AI ranks them based on how good the translation was.
  • The party with the best translation gets 7 points.
  • The party with the worst translation gets 0 points.
• They did this for 420 different language combinations. If the AI were perfectly fair, every party would end up with the same average score (about 3.5 points), just like a coin flip.

4. The Findings: The "VIP Treatment"

The results were shocking. The AI was not a neutral referee. It was a biased fan.

• The Favorites: The "Establishment" parties (like the European People's Party and the Socialists) got the VIP treatment. Their speeches were translated with high accuracy and smoothness.
• The Underdogs: The "Outsider" parties (like the far-left or non-aligned members) got the rough treatment. Their speeches were translated with more errors and lower quality.
• The Pattern: This happened across all the different AI models they tested, regardless of how big or smart the model was. It wasn't just one bad robot; it was a systemic issue.

Think of it like this: If you walk into a restaurant, the waiter might serve the steak to the regular customers perfectly, but accidentally drop the fries for the new customers. The food is the same, but the service depends on who you are.

5. Why Does This Happen?

The authors suggest a few reasons, like a recipe that was cooked with too much of one ingredient:

1. Training Data: The AI learned from the internet. Maybe there are more high-quality translations of mainstream party speeches online than there are for smaller, fringe parties.
2. Topic Differences: Mainstream parties might talk about standard topics (economy, health) that are easier for AI to translate, while outsiders might use more complex or niche language.

6. The Big Warning

The paper ends with a serious note of caution.

• The Stakes: We are starting to use AI to help translate laws, debates, and news in democracies.
• The Risk: If an AI translates a speech from a mainstream politician perfectly but garbles the speech of a minority politician, it distorts democracy. It makes the mainstream look smarter and the outsiders look confused or incoherent.

The Takeaway

This paper is a wake-up call. It shows that AI isn't just a neutral tool; it carries the biases of the data it was trained on. Just because a robot speaks 21 languages doesn't mean it treats all speakers equally. If we want fair democracies, we need to make sure our translators are fair referees, not biased fans.

Technical Summary

Here is a detailed technical summary of the paper "Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset."

1. Problem Statement

Current assessments of political biases in Large Language Models (LLMs) predominantly rely on simulated English surveys (e.g., asking models to agree/disagree with political statements). This approach has significant limitations:

• Monolingual Bias: It focuses almost exclusively on English, ignoring how biases manifest in multilingual contexts.
• Prompt Brittleness: Survey results are highly sensitive to phrasing and often lack coherence.
• Limited Scope: It fails to capture the diverse political spectrum of the European Union (e.g., Green-Alternative-Libertarian vs. Traditional-Authoritarian-Nationalist), which differs from the US binary liberal-conservative axis.

The authors propose an alternative framework: evaluating political fairness through Machine Translation (MT) quality. The core hypothesis is that if an LLM is politically biased, it will systematically translate speeches from certain political parties with higher or lower quality than others, regardless of the source language.

2. Key Contributions

A. The 21-EuroParl Dataset

The authors introduce 21-EuroParl, a novel, large-scale, 21-way multiparallel dataset derived from European Parliament (EP) proceedings (2009–2011).

• Scale: 72,234 instances, 1.5 million sentences, 40 million words, covering 249 million characters.
• Languages: 21 EU languages (bg, cs, da, de, el, en, es, et, fi, fr, hu, it, lt, lv, nl, pl, pt, ro, sk, sl, sv).
• Metadata: Unlike previous datasets (e.g., EuroParl v10, LinkedEP), this dataset includes granular metadata at the sentence level, specifically:
  • Speaker identity.
  • Political Affiliation: Both National Party and EU Party (8 major groups: EPP, S&D, ALDE, ECR, EFD, NGL, EFA, NA).
  • EU Committee and Country.
• Alignment: Strict sentence-level alignment across all 21 languages (420 language pairs), filtered to ensure 1-to-1 alignment quality (Bertalign score > 0.8).

B. Methodological Innovation: Borda Count Aggregation

Standard MT metrics (BLEU, COMET) are not well-calibrated across different language pairs, making direct score comparison impossible. To assess fairness, the authors developed a Borda Count aggregation method:

1. Ranking: For a specific language pair and model, all 8 EU parties are ranked based on their translation scores (sBLEU or COMET).
2. Scoring: The party with the lowest score receives 0 points, the next lowest 1 point, up to 7 points for the best-performing party.
3. Aggregation: These points are summed across all 420 language pairs.
4. Fairness Baseline: In a perfectly fair model, every party would receive an average score of 3.5 (random ranking). Significant deviations indicate systematic bias.

3. Experimental Setup

• Models Tested: Five post-trained LLMs across three families:
  • Gemma-3: 4B parameters.
  • Qwen3: 4B and 8B parameters.
  • Llama-3.1: 8B and 70B parameters.
  • (Note: Smaller Llama-3.2 models were excluded due to poor translation performance).
• Task: Zero-shot translation across all 420 language pairs using a standardized prompt template.
• Metrics: sBLEU (sentence-level BLEU) and COMET (neural metric).
• Control: The study controlled for confounding variables like "Topic" (using EU Committee metadata) and "Original Language" (translationese effects) using two-way ANOVA.

4. Key Results

A. Political Fairness Findings

The study reveals systematic political bias across all model families and sizes. The bias is not a simple "left vs. right" gradient but favors majority parties over outsider/minority parties.

• Favored Parties (High Translation Quality):
  • EPP (European People's Party - Center-right)
  • S&D (Socialists & Democrats - Center-left)
  • ALDE (Alliance of Liberals and Democrats)
  • Result: EPP often achieved near-perfect Borda scores (~7.0), meaning it was consistently ranked first.
• Disadvantaged Parties (Low Translation Quality):
  • NGL (European United Left/Nordic Green Left)
  • NA (Non-attached Members)
  • Result: NGL consistently ranked last (Borda scores near 0.0).
• Consistency: The bias pattern was highly consistent across model sizes (8B vs. 70B) and families (Llama vs. Qwen vs. Gemma), with Pearson correlations of Borda counts exceeding 0.96.
• Statistical Significance: A Kruskal-Wallis test confirmed that the "Party" variable significantly explains performance variance in the vast majority of language pairs (p < 0.01), even after controlling for topic and original language.

B. Language Fairness Findings

The models also exhibited significant language bias:

• Best Translated: English (en) and Spanish (es).
• Worst Translated: Lithuanian (lt) and Estonian (et).
• Surprise: Despite their status as official working languages of the EU, German (de) and French (fr) ranked lower than Spanish, Bulgarian, and Romanian in some metrics.

5. Significance and Implications

• New Evaluation Paradigm: The paper establishes that translation quality is a viable and robust proxy for measuring political bias in multilingual LLMs, bypassing the brittleness of survey-based methods.
• Democratization Risk: The findings suggest that LLMs may systematically amplify the voices of established, mainstream political groups while degrading the quality of translation for fringe or minority political groups. This poses a risk to multilingual democratic deliberation (e.g., the "Conference on the Future of Europe"), where accurate translation is essential for fair participation.
• Data Origin: The bias likely stems from the pre-training or post-training data, which appears to over-represent the discourse of major parties (EPP, S&D) compared to smaller factions.
• Resource Availability: The release of 21-EuroParl provides the community with a critical resource for studying multilingual fairness, political NLP, and machine translation bias, filling a gap left by previous biparallel or metadata-poor datasets.

Conclusion

The authors demonstrate that multilingual LLMs are not politically neutral tools. They exhibit a systematic preference for translating the speech of mainstream, majority political parties with higher fidelity than that of outsider parties. This "political unfairness" in translation suggests that deploying these models in democratic processes without mitigation could inadvertently distort political discourse and marginalize minority viewpoints.