💬 NLP

Framing Political Bias in Multilingual LLMs Across Pakistani Languages

This paper presents a systematic evaluation of 13 state-of-the-art Large Language Models across five Pakistani languages, revealing that while these models generally reflect Western liberal-left biases, they exhibit distinct, language-conditioned authoritarian framing in regional tongues, thereby highlighting the urgent need for culturally grounded, multilingual bias auditing frameworks.

Original authors: Afrozah Nadeem, Mark Dras, Usman Naseem

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Afrozah Nadeem, Mark Dras, Usman Naseem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, well-read robots (Large Language Models, or LLMs) that have read almost everything written in English. They are great at answering questions, writing stories, and debating topics. However, most of what they've read comes from Western countries like the US and UK.

This paper is like a cultural reality check for these robots. The researchers asked: "Do these robots understand the political and cultural nuances of Pakistan, or do they just force Western ideas onto Pakistani languages?"

Here is a breakdown of their findings using simple analogies:

1. The Problem: The "One-Size-Fits-All" Suit

Think of these AI models as tailors making suits. They have a perfect suit made for English speakers (Western context). But when they try to make a suit for someone speaking Urdu, Punjabi, Sindhi, Pashto, or Balochi, they just shrink the English suit down.

The researchers found that while the robots are generally "liberal" (progressive/left-leaning) when speaking English, they start acting differently when they switch to Pakistani languages.

The Analogy: It's like a person who is very open-minded and free-spirited at a party in London, but suddenly becomes very strict and follows rigid rules when they walk into a traditional village in Pakistan. The robot isn't changing its mind; it's just reacting to the "language costume" it's wearing.

2. The Experiment: The "Political Compass" Test

To test this, the team used a famous test called the Political Compass Test (PCT). Imagine a giant map with four corners:

Top-Left: Liberal/Left (Free economy, progressive social views)
Top-Right: Conservative/Right (Free economy, traditional social views)
Bottom-Left: Authoritarian/Left (State control, progressive social views)
Bottom-Right: Authoritarian/Right (State control, traditional social views)

They translated 62 questions from this test into five Pakistani languages and asked 13 different AI models to answer them. They also asked the models to write news headlines about 11 sensitive topics (like the death penalty, blasphemy laws, and minority rights).

3. The Findings: The "Language Switch" Effect

The results were surprising and showed a clear pattern:

The English Baseline: When the robots spoke English, they mostly landed in the Libertarian-Left corner. They sounded very Western, progressive, and focused on individual freedom.
The Pakistani Shift: When the same robots spoke Urdu, Punjabi, or Sindhi, they shifted.
- Some became more Authoritarian-Left. This means they started sounding like they supported strong government control and state responsibility, which is a common theme in local political discourse.
- The "Death Penalty" Example: The paper highlights a specific case where a robot, when asked about the death penalty in Urdu, used religious and cultural phrases. A Western-trained detector might have misinterpreted this as "supporting violence" because it didn't understand the cultural context. In reality, the robot was reflecting a local perspective that the Western detector couldn't "read."

4. The "Framing" Analysis: How They Say It

The researchers didn't just look at what the robots said (their stance); they looked at how they said it (their framing).

The News Headline Test: They asked the robots to write headlines for and against various issues.
- The Result: The robots used different "lenses" depending on the language.
- Example: When discussing minority rights, some robots focused on institutions (like the Supreme Court or the State), sounding very official and hierarchical. Others focused on individual stories and human rights.
- The Tone: For sensitive topics like blasphemy or abortion, the robots sounded very negative and divided. For topics like education, they sounded hopeful.

5. The Solution: "Cultural Tuning"

The paper also tested what happens if you take a robot and "fine-tune" it specifically for Urdu (teaching it more about local culture).

The Analogy: It's like taking a student who only studied in London and sending them to a local university in Pakistan for a semester.
The Result: These "culturally tuned" robots became much more balanced and neutral. They stopped swinging wildly between extremes and gave answers that felt more fair to the local context.

Summary of the Main Takeaway

The paper argues that language is not just a code; it carries culture.

If you use a Western-trained AI to talk about politics in Pakistan, it might accidentally distort the conversation. It might sound too "Western" or misinterpret local cultural values as bias. The researchers built a new toolkit to measure this "political bias" across different languages so that developers can build AI that respects local cultures instead of forcing a foreign worldview onto them.

In short: The paper shows that AI isn't politically neutral; its "politics" change depending on the language it's speaking, and we need to check its "political compass" in every language to ensure it's fair.

Technical Summary: Framing Political Bias in Multilingual LLMs Across Pakistani Languages

Problem Statement
While Large Language Models (LLMs) increasingly shape public discourse, evaluations of political and economic bias have predominantly focused on high-resource, Western languages. This creates critical blind spots in low-resource, multilingual regions like Pakistan, where linguistic identity is inextricably linked to political, religious, and regional ideologies. Existing bias detection tools often rely on Western political taxonomies and fail to account for the framing of ideologies in low-resource languages. Consequently, LLMs may misinterpret culturally grounded statements (e.g., traditional justice principles) as support for violence or bias, distorting civic discourse and marginalizing minority perspectives. There is a lack of systematic frameworks to audit political bias across the five major Pakistani languages: Urdu, Punjabi, Sindhi, Pashto, and Balochi.

Methodology
The authors propose a novel, culturally grounded framework that integrates ideological stance detection with multi-level narrative framing analysis. The methodology consists of three primary components:

Culturally Adapted Political Compass Test (PCT):
- The study adapts the standard 62-statement PCT, translating and culturally validating them for five Pakistani languages.
- The framework maps model responses onto a two-dimensional ideological space: Economic values (Left vs. Right) and Social values (Libertarian vs. Authoritarian).
- A two-stage computational approach quantifies stance: first, calculating a scalar score based on classifier confidence (Strongly Agree to Strongly Disagree), and second, discretizing this score into ordinal categories to project models onto the $(S_{eco}, S_{soc})$ plane.
- The dataset includes 4,030 responses generated by 13 state-of-the-art LLMs (both open and closed-source) across the five languages, verified by native speakers with a Fleiss' kappa of 0.99.
Ideological Framing Analysis:
- To capture how models express bias, the authors generate a corpus of approximately 444,340 news headlines across 11 politically salient topics (e.g., Blasphemy Laws, Death Penalty, LGBTQ Rights, Minority Rights).
- Content Framing: Headlines are classified using Boydstun's 15 cross-cutting frame dimensions (e.g., Morality, Economics, Cultural Identity) via GPT-3.5-turbo to determine dominant narrative strategies.
- Entity-Based Analysis: Named Entity Recognition (using a multilingual BERT model) extracts political figures and institutions. The study calculates entity prominence and sentiment polarity (using fine-tuned XLM-RoBERTa) to identify which actors are emphasized and how they are portrayed (positive/negative/neutral).
Experimental Setup:
- The study evaluates 13 LLMs using zero-shot prompting.
- It includes a specific analysis of "cultural adaptive" models (fine-tuned on Urdu) to compare their bias scores against standard SOTA baselines.
- Robustness is tested using five semantically equivalent but instructionally distinct prompt variants.

Key Contributions

First Large-Scale Evaluation: This is the first study to conduct a large-scale political bias evaluation across five Pakistani languages (Urdu, Punjabi, Sindhi, Pashto, Balochi).
Novel Framework: The authors introduce the first framework to combine the PCT for ideological positioning with narrative framing analysis (content, style, and entity focus) specifically for Pakistani political discourse.
Cultural Adaptation: The work provides a culturally adapted PCT with human-verified translations covering 11 topics grounded in local discourse, moving beyond generic Western instruments.
Comprehensive Dataset: The release of a gold-standard multilingual dataset comprising PCT translations and a massive headline generation corpus for bias analysis.

Results

Language-Conditioned Ideological Modulation: While LLMs predominantly reflect liberal-left orientations consistent with Western training data in English, they exhibit significant shifts in Pakistani languages. For instance, GPT models tend to adopt libertarian-left positions in English but shift toward authoritarian-left or authoritarian-right stances in regional languages like Urdu and Punjabi.
Authoritarian Framing in Regional Languages: Models exhibit more authoritarian framing in regional languages compared to English, highlighting that linguistic choice activates culturally specific ideological shifts.
Model-Specific Patterns:
- Claude consistently demonstrates the strongest libertarian stance across languages.
- GPT-4-turbo leans most economically left.
- Open-source models (e.g., Mistral, DeepSeek) generally remain more liberal-libertarian, whereas fine-tuned Urdu models showed a tendency to shift toward authoritarian-right stances in some contexts.
- BERT variants tended to lean right across languages.
Framing and Sentiment:
- Models exhibit varying uses of "Cultural Identity" frames when discussing religious minorities. Authoritarian-leaning models tend to frame minority rights through institutional and geographic hierarchies (emphasizing the State), while libertarian-leaning models incorporate broader legal frameworks and regional experiences.
- Sentiment analysis reveals that sensitive topics like Blasphemy Laws and the Death Penalty elicit overwhelmingly negative sentiment, whereas topics like Education and Language Policy show more positive or hopeful tones.
Impact of Fine-Tuning: Culturally adapted (fine-tuned) models produced more balanced, centered responses with lower bias scores compared to SOTA baselines, suggesting that cultural-to-linguistic adaptation can promote political neutrality.

Significance
The paper argues that political bias in LLMs is not merely a translation issue but a complex interaction between linguistic context and cultural ideology. The findings demonstrate that Western political frames can mischaracterize local thought, potentially amplifying inequality for speakers of low-resource languages. By bridging the PCT with narrative framing dimensions, the study offers a reproducible methodology for diagnosing political bias in multilingual LLMs. The authors conclude that building fair, inclusive AI systems for linguistically diverse societies requires culturally grounded evaluation frameworks that move beyond Western-centric assumptions to ensure equitable and responsible AI deployment.