Fluent but Foreign: Even Regional LLMs Lack Cultural Alignment

Technical Summary: "Fluent but Foreign: Even Regional LLMs Lack Cultural Alignment"

Problem Statement
While Large Language Models (LLMs) are rapidly adopted globally, they predominantly encode Western-centric values, beliefs, and knowledge. In response, developers in the Global South are building "regional" or "sovereign" LLMs trained on local languages to serve specific communities. However, a critical gap exists in evaluating whether these models are merely linguistically fluent or truly culturally aligned. Current multilingual benchmarks focus on reasoning, classification, and generation capabilities but fail to assess alignment with local cultural values, practices, and lived realities. This paper investigates whether regional training actually endows LLMs with the cultural alignment required to serve their target populations, using India as a case study.

Methodology
The authors employ a "thick × wide" evaluation framework grounded in Hofstede's cultural onion, assessing two dimensions: values (what people believe) and practices (what people do). The study evaluates 12 models: six Indic models (e.g., nanda-10b, sarvam-m-24b) and six global baselines (e.g., GPT-4o, Llama-3, Mistral).

The evaluation consists of four benchmark tasks and one user study:

1. Cultural Map (Values): Using the World Values Survey (WVS), models are positioned on the Inglehart–Welzel cultural map (Traditional vs. Secular-Rational; Survival vs. Self-Expression) to compare their value orientation against national averages.
2. Opinion Alignment (Values): Using GlobalOpinionQA (derived from Pew Global Attitudes), the study measures how closely model predictions match the response distributions of Indian versus American publics on 94 divergent questions. A "Cultural Alignment Differential" (CAD) is calculated to quantify bias.
3. Cultural Knowledge (Practices): Using a subset of CulturalBench, models answer multiple-choice questions on factual customs and etiquette for India and the US.
4. Cultural Adaptation (Practices): Using NormAd, models are tested on their ability to apply cultural norms in situated scenarios (e.g., dining etiquette) under varying levels of context (Country, Value+Country, Rule-of-Thumb).
5. User Study (Downstream Application): A study with 115 Indian participants involved writing essays on culturally grounded topics (festivals, food, public figures) with AI assistance. The study compared writing suggestions from a top-performing Indic model (Sarvam-M) against a global model (GPT-4o) to measure cultural homogenization, exoticization, and writing style preservation.

The study also tested various prompting strategies (Demographic, Cross-lingual, Hindi translation) to see if they could recover cultural alignment.

Key Results

• Misalignment with Local Values: Across all tasks, Indic models did not align better with Indian norms than global models. In the Cultural Map analysis, no Indic model fell into the quadrant representing India; in some cases, an average American respondent was a closer proxy for Indian values than any Indic model tested.
• Opinion Bias: On GlobalOpinionQA, several Indic models showed a statistically significant tilt toward US public opinion rather than Indian opinion. Normalizing for the degree of disagreement between the two populations revealed that this US bias was even more pronounced.
• Knowledge Deficits: Indic models performed significantly worse on Indian cultural knowledge questions than on US questions. Regional fine-tuning often failed to inject new cultural knowledge and, in several cases (e.g., gajendra-7b, sarvam-m-24b), degraded performance on both Indian and US cultural questions compared to their base global models.
• Failure in Adaptation: In NormAd, Indic models struggled to apply Indian social norms, performing near random chance in implicit reasoning contexts, while global models performed better on US norms.
• Writing Homogenization: In the user study, suggestions from the Indic model (Sarvam-M) led to essays that were only marginally more similar to natural Indian writing than those from GPT-4o. The Indic model frequently suggested Westernized artifacts (e.g., barbecues for Independence Day) and exoticized descriptions, failing to preserve authentic Indian cultural expressions.
• Ineffectiveness of Prompting: Prompting strategies (including demographic cues and language translation) yielded negligible gains in cultural alignment and could not recover the underlying misalignment.

Key Contributions

1. Multi-dimensional Evaluation Framework: The paper operationalizes a "thick × wide" approach, combining population-scale survey data (WVS, Pew) with community-grounded QA datasets and a downstream user study to evaluate cultural alignment.
2. Empirical Evidence on Regional Training: This is the first large-scale empirical study demonstrating that regional fine-tuning on local languages does not necessarily result in cultural alignment. It reveals that regional models often reflect Western worldviews despite their linguistic localization.
3. Bridging NLP and HCI: The work positions cultural alignment as a first-class design requirement, arguing that linguistic fluency is insufficient for the equitable design of AI systems in non-Western contexts.

Significance and Claims
The paper claims that the current paradigm of building "sovereign" LLMs is flawed if it relies solely on translating Western-centric pre-training data or fine-tuning on thin slices of local language data. The authors argue that the bottleneck lies in the scarcity of culturally grounded data, particularly for pre-training.

The significance of this work lies in its challenge to the assumption that language localization equates to cultural competence. The authors call for:

• Native, community-authored corpora to replace translated or Western-centric data.
• Thick × wide evaluation protocols that pair community co-design with population-scale data to detect cultural brittleness before models are scaled.
• Community stewardship of data resources to ensure AI systems reflect local values and lived realities rather than imposing a Western gaze.

Ultimately, the paper concludes that without addressing the data imbalance and rethinking training objectives, regional models risk perpetuating cultural hegemony under the guise of localization.