Large Language Models are Perplexed by some Political Parties

Technical Summary: Large Language Models are Perplexed by some Political Parties

Problem Statement
While Large Language Models (LLMs) are increasingly deployed in political applications, their political fairness remains under-studied. Existing literature primarily assesses "political biases" in Instruction-Tuned (IT) LLMs using survey-based methods (e.g., asking models to answer political compass questions). These methods often suffer from brittleness regarding prompt phrasing and are generally inapplicable to base LLMs. Furthermore, prior work on translation fairness (Lerner and Yvon, 2025) suggests that translation quality varies by the political affiliation of the speaker, but this relies on parallel corpora and downstream metrics. The authors posit that a fair model should assign equal probability (perplexity) to texts regardless of their partisan content, yet it is unknown if base LLMs exhibit systematic unfairness in their pretraining distributions across diverse political groups and languages.

Methodology
The authors propose a method to assess political fairness using perplexity (and related information-theoretic metrics like Negative Log-Likelihood, Bits Per Character, and Bits Per English Character) on monolingual corpora. The core hypothesis is that a fair model should exhibit equal perplexity for texts from all political groups.

• Datasets: The study utilizes three datasets covering 37 languages:
  1. Manifesto: Political party programs (2018–present).
  2. Parlamint: National parliamentary debates (2022–present).
  3. 21-EuroParl: A multi-parallel corpus of European Parliament proceedings (2009–2011) aligned at the sentence level across 21 languages.
• Models: Experiments cover ten LLMs from three families (Gemma-3, Qwen3, Llama-3.1) in two sizes (4B/8B and 8B/70B), evaluating both Base LLMs and their Instruction-Tuned (IT-LLM) counterparts.
• Metrics & Analysis:
  • Fairness Assessment: The authors compute perplexity (PPL), BPC, and BPEC on texts stratified by political party. To determine if perplexity varies significantly by party, they employ Kruskal-Wallis analysis (a non-parametric ANOVA).
  • Aggregation: For cross-language comparison, they use Borda count aggregation to rank political parties or families (e.g., Social Democratic vs. Nationalist) based on their perplexity scores.
  • Correlation: To link pretraining fairness with downstream performance, they compute the Spearman correlation between the base LLM's perplexity and the downstream translation metrics (sBLEU and COMET) of the corresponding IT-LLM.

Key Results

1. Systematic Unfairness in Pretraining: Across ten LLMs and datasets covering 37 languages, LLMs are consistently more perplexed by the texts of far-right and nationalist parties compared to social-democratic parties.
   • In the Portuguese case study, radical-left and far-right parties had the highest (worst) NLL, while the left-wing Socialist party had the lowest (best) NLL.
   • Aggregated across 10 languages in the Manifesto dataset, Nationalist/Radical Right parties consistently received the worst Borda scores (lowest ranking), while Social Democratic parties received the best.
2. Base vs. Instruction-Tuning: The political unfairness observed in base LLMs is highly correlated with that of their instruction-tuned counterparts (Pearson $r \approx 0.99$). The study finds that instruction-tuning does not significantly alter the underlying political fairness established during pretraining.
3. Correlation with Downstream Translation: There is a strong negative correlation between the perplexity of a political party's text in a base LLM and the translation quality (sBLEU/COMET) of that text when generated by the IT-LLM.
   • Parties with higher perplexity (lower probability assigned by the model) receive lower translation scores.
   • This correlation holds even when using unconditioned perplexity (without source text context), suggesting that the model's intrinsic difficulty in modeling a specific political group's language directly impacts its ability to generate or translate that language later.

Key Contributions

• Perplexity as a Fairness Metric: The paper introduces a method to assess political fairness using perplexity on plain monolingual corpora, applicable to both base and instruction-tuned models without requiring reference translations or survey prompts.
• Pretraining Origin of Bias: The authors demonstrate that political fairness issues stem primarily from the pretraining phase, as base LLMs and their IT counterparts show nearly identical fairness profiles.
• Multilingual Consistency: The findings hold across 37 languages and 420 language pairs, suggesting that political features are shared across languages within a single model rather than being language-specific subspaces.
• Prediction of Downstream Performance: The study establishes that base model perplexity can predict downstream translation fairness, offering a way to estimate translation bias without parallel data.

Significance and Claims
The paper claims that the political fairness of LLMs is not a gradual left-right spectrum but rather a specific disadvantage for certain groups (particularly far-right and nationalist parties) compared to social-democratic ones. The authors argue that because this bias originates in pretraining, standard instruction-tuning alignment methods are insufficient to correct it. Consequently, the work calls for:

• Alternative post-training alignment methods.
• The design of fair pretraining methods.
• A broader reflection on integrating LLMs into sensitive political applications.

The study contrasts with previous survey-based literature by focusing on the model's intrinsic probability distributions rather than its generated answers to prompts, providing evidence that "political bias" is a fundamental property of the model's training distribution.