Measuring Political Stance and Consistency in Large Language Models

Technical Summary: Measuring Political Stance and Consistency in Large Language Models

Problem Statement
The rapid adoption of Large Language Models (LLMs) for information retrieval has introduced significant risks regarding political bias and the reliability of normative claims. Unlike factual queries, political questions often lack a single "correct" answer, making it difficult for users to fact-check model outputs. LLMs may inadvertently reflect biases present in their training data or acquire new biases during the alignment process. While prior research has examined political leanings in specific domains (e.g., U.S. elections, gun control), there is a lack of systematic analysis regarding widely debated regional and geopolitical issues, such as the Russia–Ukraine war, the Palestine-Israel conflict, and policies in China and the Middle East. Furthermore, the consistency of these stances under varying prompting strategies remains underexplored.

Methodology
The authors conducted a comprehensive evaluation of nine LLMs across 24 politically sensitive issues, with a focus on topics from Asia, the Middle East, and Eastern Europe. The selected models include Mistral-7B, Ollama-8B, DeepSeek, Grok-3-mini, Gemini-2.5-Flash, o3-mini, GPT-4o-mini, GPT-4o, and GPT-5. The study employed five distinct prompting strategies to assess stance consistency:

1. Direct Questions: Asking for opinions without prior context.
2. Providing Opposite Arguments: Presenting an argument opposing the model's inferred baseline stance.
3. Providing Arguments for Both Sides: Supplying arguments for both conflicting parties.
4. Changing Question Formulation: Reversing the framing of the question (e.g., asking if an action is "justified" vs. "unjustified").
5. Prompting in Different Languages: Translating prompts into the local languages of the countries involved in the dispute (e.g., Turkish and Greek for the Aegean Islands dispute) to test for language-induced bias.

For each strategy, three paraphrased variants were generated to ensure robustness. Model responses were analyzed using an automated "judge" model (Ollama-8B), with Grok-3-mini serving as an external validator to minimize self-evaluation bias. Human validation on a random sample confirmed the judge's accuracy at 96.7%.

Key Contributions

• Expanded Scope: The study analyzes political leanings on 24 issues largely absent from prior literature, including the Qatar blockade, the oppression of Palestinians, the Uyghur situation, and various territorial disputes.
• Consistency Analysis: The authors investigate how five different prompting strategies influence the stability of model stances, moving beyond single-prompt evaluations.
• Reproducibility: The authors have released their code and data to support further research in this domain.

Results

• General Stance: In almost all cases, LLMs adopted a specific side on sensitive issues rather than remaining neutral.
• Consensus: For 8 out of 24 topics, all nine models shared the same political leaning (e.g., support for Palestinian rights, criticism of the Gulf blockade on Qatar, and support for Ukraine in the Russia-Ukraine war).
• Model Variability:
  • Grok-3-mini emerged as the most consistent model, changing its stance on only five issues across all prompting strategies.
  • Mistral-7B was the least consistent, shifting its stance in 19 out of 24 cases.
  • DeepSeek (developed by a Chinese company) consistently generated output supporting the Chinese government on relevant issues (IDs 11–14), whereas other models largely took the opposite stance.
• Prompting Effects:
  • Language Sensitivity: Changing the prompt language was the most effective method for altering stances on issues involving countries with different languages. In 36.7% of these cases, models reversed their stance based on the language used (e.g., supporting Greece when prompted in Greek and Turkey when prompted in Turkish).
  • Stability: Stances on the Qatar blockade and the oppression of Palestinians remained unchanged across all models and all prompting methods.
  • Total Variability: Out of 216 total model-issue combinations (24 issues × 9 models), stances remained stable in 118 cases, indicating substantial consistency overall, though significant variability exists.

Significance and Claims
The paper claims that these findings are essential for raising user awareness regarding the limitations of LLMs in political guidance. The authors argue that users should interpret model outputs critically, recognizing that stances can be malleable depending on the prompt's language and framing. For developers, the study highlights the need to address pre-existing biases in training data and alignment processes. The authors suggest that future safeguards could include mechanisms requiring models to provide evidence-backed justifications and counterarguments to help users form their own conclusions. The study positions itself as a necessary step toward socially beneficial information retrieval, acknowledging that while malicious actors could theoretically exploit these findings, the primary goal is to foster transparency and critical engagement with AI-generated political content.