Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion

This paper presents a multilingual audit revealing that while contemporary large language models generally align with public opinion on broad social issues across Asian regions, they consistently fail to accurately represent diverse religious viewpoints—particularly those of minority groups—and often amplify negative stereotypes, a problem that persists despite lightweight prompting interventions and remains undetected by standard bias benchmarks.

The Gist

Here is an explanation of the paper "Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion," translated into simple language with some creative analogies.

🌍 The Big Picture: The "Global Translator" Who Misses the Mark

Imagine you have a super-smart, all-knowing robot librarian (a Large Language Model, or LLM) that has read almost everything on the internet. You ask it, "What do people in India, Japan, or Thailand think about religion?"

You expect the robot to give you an answer that sounds like a real conversation with a local person. But this paper argues that the robot is actually more like a tourist who has only read guidebooks written in English. Even if you ask the robot in Hindi, Thai, or Korean, it often answers with the "Western" or "English-speaking" worldview it learned from its training data.

The researchers found that while these robots are great at general topics (like politics or economics), they get stuck in a cultural rut when it comes to religion, often misrepresenting minority groups and amplifying stereotypes.

---

🔍 The Experiment: The "Opinion Poll" Test

The researchers wanted to see if these AI models actually "speak" the culture of the people they are talking to.

The Setup:

1. The Ground Truth: They took real, massive surveys conducted by the Pew Research Center. These surveys asked thousands of real people in 12 Asian countries (like India, South Korea, Thailand) about their religious beliefs and social views. This is the "real answer key."
2. The Test: They asked the same questions to top AI models (like GPT-4o-Mini, Gemini, Llama, etc.).
3. The Twist: They asked the questions in English AND in the local languages (like Hindi, Japanese, Thai) to see if speaking the local language helped the AI understand better.

The Analogy:
Imagine a chef (the AI) trying to cook a traditional dish for a family.

• The Ground Truth is the family's actual recipe and taste preferences.
• The AI is the chef who has only cooked Western-style food before.
• The researchers asked: "If you ask the chef to cook this dish in the local language, will the taste change to match the family's preference?"

---

🚨 The Findings: Where the Robot Fails

1. The "Religion Blind Spot"

The AI models were surprisingly good at guessing what people thought about general things (like "Is the economy good?"). But when the topic turned to religion, the AI got it wrong.

• The Problem: The AI tended to favor the "majority" or "Western" view and often got the views of minority religious groups completely wrong.
• The Metaphor: It's like a radio station that plays the top 40 hits perfectly but gets the local folk music completely wrong, often playing a distorted, stereotypical version of it.

2. The "Language Illusion"

The researchers hoped that if they asked the AI in the local language (e.g., "What do Muslims in India think?" in Hindi), the AI would suddenly "wake up" and understand the local culture.

• The Result: It helped a little bit, but not enough.
• The Metaphor: It's like putting a French accent on a person who doesn't actually know French. They might sound slightly more local, but they still don't understand the deep cultural nuances. The AI's "brain" was still wired with English-centric data.

3. The "Stereotype Amplifier"

When the researchers tested the AI on specific bias benchmarks (like asking if a negative statement about a religious group sounds "plausible"), the AI often said yes to negative stereotypes.

• The Finding: The AI was more likely to believe that negative things about minority groups (like Shia Muslims or Jains) were true, compared to positive things.
• The Metaphor: The AI is like a gossip columnist who has read too many sensationalist tabloids. It assumes the worst about certain groups because that's what it saw most often in its training data.

---

🛠️ Why Does This Happen? (The Root Causes)

The paper suggests three main reasons for this "cultural gap":

1. The Training Diet: The AI was fed a diet of internet data that is mostly English and Western. It's like feeding a panda only bamboo from California; it might survive, but it won't taste like the bamboo from its native home.
2. The "Safety" Filter: When companies try to make AI "safe," they often use feedback from Western users. This accidentally creates a filter that blocks or distorts the views of non-Western minorities.
3. The "Black Box" Problem: Most people use these AIs through an API (a black box). They can't see inside the code to fix the bias. They can only try to "prompt" (ask) the AI differently, which is like trying to fix a broken engine by shouting instructions at the hood of the car.

---

💡 What Can We Do? (The Takeaway)

The paper concludes that simply making AI "multilingual" (able to speak many languages) is not enough. We need "multicultural" AI.

• The Solution: We need to audit these models specifically for different regions and cultures before we let them loose on the world.
• The Future: We need to train these models on data that actually represents the local people, not just the global internet. We need to give the "chef" the real local recipe, not just a translation of a Western cookbook.

In a nutshell:
These AI models are powerful tools, but right now, they are cultural tourists who haven't learned the local customs. If we don't fix this, they risk spreading stereotypes and misunderstanding the very people they are supposed to help, especially in the diverse and religiously complex societies of Asia.

Technical Summary

Here is a detailed technical summary of the paper "Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion."

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in multilingual, multicultural environments, yet they rely heavily on English-centric training data. This creates a risk of cultural misalignment, where models fail to reflect the diverse values of non-Western societies, particularly in sensitive domains like religion.

• The Gap: While existing alignment research focuses on Western (US/European) contexts and English prompts, there is a lack of systematic auditing for Asian populations.
• The Specific Challenge: Religion remains a central, politically significant aspect of society in many Asian nations, unlike in many Western contexts. Current models may inadvertently amplify negative stereotypes or misrepresent minority religious viewpoints due to skewed training corpora and alignment processes (RLHF) that favor majority or Western norms.
• Core Question: Do contemporary LLMs accurately represent public opinion on sensitive religious topics in Asia, and does prompting in local languages mitigate these biases?

2. Methodology

The authors propose a comprehensive, multilingual audit framework comparing model-generated opinion distributions against ground-truth human survey data.

A. Ground Truth Data

• Source: Pew Research Center surveys (Pew-Templeton Global Religious Futures Project) covering 12 countries/territories across India, East Asia, and Southeast Asia.
• Surveys: Religion in India, Religion and Views of an Afterlife in East Asia, and Buddhism, Islam and Religious Pluralism in South and Southeast Asia.
• Preprocessing: Survey questions (originally in English metadata) were translated into local languages (e.g., Hindi, Sinhala, Thai, Korean, Vietnamese) via a high-fidelity, crowd-sourced manual translation pipeline to ensure semantic and cultural accuracy.
• Weighting: Human responses were aggregated using statistical weights provided by Pew to ensure national representativeness.

B. Models Evaluated

• Closed-Source: GPT-4o-Mini, Gemini-2.5-Flash.
• Open-Weight: Llama 3.2, Mistral 7B, Gemma 3.

C. Evaluation Metrics

The study measures "representativeness" by comparing the Model Opinion Distribution ($D_M$) against the Human Opinion Distribution ($D_O$) using:

1. Jensen-Shannon Divergence (JSD): Measures the similarity between two probability distributions.
2. Hellinger Distance (HD): Another metric for distributional similarity.
3. Wasserstein Distance (WD): Used to calculate a "Representativeness Score" ($R_M$), accounting for the ordinal nature of answer choices.
   • Formula: $R_M(Q) = 1 - \frac{WD(D_M, D_O)}{N-1}$, where higher scores indicate better alignment.

D. Bias Benchmarks

To assess concrete downstream harms, the authors evaluated models on four culturally aware benchmarks:

• CrowS-Pairs: Measures stereotyping tendencies via minimal pairs.
• IndiBias: Tests biases on Indian identity axes (religion, caste, etc.) in English and Hindi.
• ThaiCLI: Evaluates alignment with Thai cultural norms (Royal family, religion, politics) via chosen/rejected answer pairs.
• KoBBQ: Korean Bias Benchmark for QA, testing disambiguation and bias in Korean contexts.

E. Intervention Strategies

The study tested "lightweight interventions" to see if they could reduce gaps:

• Demographic Priming: Prefixing prompts with context (e.g., "You are a citizen of...").
• Native Language Prompting: Switching prompts from English to local languages.

3. Key Results

A. General Representativeness vs. Religious Topics

• Broad Alignment: Models generally perform well on non-religious social issues (governance, demographics), achieving representativeness scores ($R_M$) above 94%.
• Religious Misalignment: Performance drops significantly on religion-related prompts.
  • GPT-4o-Mini: Drops from ~95.2% (non-religious) to ~90.2% (religious).
  • Gemini-2.5-Flash: Drops from ~94.6% to ~89.9%.
• Minority Groups: Models consistently fail to represent minority religious viewpoints accurately, often amplifying negative stereotypes. For example, in IndiBias, negative framings of Shia and Sunni Muslims were judged as more plausible than positive ones (high $\Delta$ELO scores).

B. Impact of Local Language Prompting

• Partial Mitigation: Switching to local languages consistently reduces the Jensen-Shannon Divergence (improving distributional overlap).
  • Example: Gemma-3 in Sri Lanka (Sinhala) saw a ~31% reduction in divergence compared to English.
• Limitations: The Hellinger Distance (measuring fundamental probability shifts) remained largely resistant to language changes. This suggests that while local language cues help models focus probability mass, they do not fully correct the underlying biased worldview encoded in the model weights.
• Failure Case: In some instances (e.g., Llama 3.2 in Taiwan), local language prompting did not significantly improve alignment, indicating deep-seated training data biases that translation cannot fix.

C. Benchmark Findings

• CrowS-Pairs: GPT-4o-Mini showed robust anti-stereotype performance (92% accuracy) across languages. Gemini-2.5-Flash showed higher bias rates (16%) and more invalid responses, particularly in Vietnamese.
• ThaiCLI: GPT-4o-Mini achieved high cultural sensitivity scores (>8.3/10) on sensitive topics like religion and the royal family.
• KoBBQ: Disambiguating prompts significantly improved accuracy (0.61 $\to$ 0.96) and reduced bias in Korean identity benchmarks, highlighting the importance of prompt specificity.

4. Key Contributions

1. Multilingual Audit Framework: Introduced a rigorous methodology for auditing LLM cultural alignment across 12 Asian countries using native-language translations of high-quality survey data.
2. Religion as a Critical Lens: Demonstrated that religion is a specific domain where LLMs consistently fail to align with public opinion, even when they perform well on general social issues.
3. Quantification of Interventions: Provided empirical evidence that while native language prompting reduces distributional divergence, it is insufficient to eliminate fundamental representational harms or correct deep-seated stereotypes.
4. Bias Benchmarking: Integrated diverse regional benchmarks (IndiBias, ThaiCLI, KoBBQ) to show how high-level distributional gaps manifest as concrete representational harms (e.g., normalizing negative framings of minority groups).

5. Significance and Implications

• Beyond Translation: The paper argues that multilingual capability does not equate to cultural representativeness. Models can be fluent in a language but still reflect the values of their dominant (Western/English) training data.
• Risk of Harm: The misalignment in religious domains poses a risk of reinforcing prejudices, marginalizing minority groups, and polarizing discourse in societies where religion is politically central.
• Need for Systemic Change: The authors conclude that lightweight interventions (prompting) are insufficient. True alignment requires:
  • Data-Centric Solutions: Pre-training on targeted, locally sourced data (native narratives, local journalism).
  • Architectural Shifts: Moving beyond black-box APIs to allow for activation engineering and fine-tuning on culturally specific corpora.
  • Regionally Grounded Audits: Establishing systematic, region-specific evaluation protocols before global deployment of LLMs.

In summary, the paper highlights a critical "Mind the Gap" between the global deployment of LLMs and the diverse cultural realities of Asian societies, urging a shift from superficial translation to deep, data-driven cultural alignment.