Technical Summary: Camellia – Benchmarking Cultural Biases in LLMs for Asian Languages

Problem Statement

As Large Language Models (LLMs) integrate into global technology, their ability to handle culturally diverse entities—such as names, locations, and food—without bias is critical. Prior research, specifically by Naous et al. (2024), demonstrated that LLMs exhibit a preference for Western-associated entities over Arabic ones. However, due to a lack of entity-centric multilingual benchmarks, it remains unclear whether these biases manifest across other non-Western languages. The paper identifies a gap in evaluating how LLMs distinguish between entities native to specific Asian cultures versus those associated with Western culture, and how these biases affect model performance in tasks like sentiment analysis and information extraction.

Methodology

The authors introduce Camellia (Cultural Appropriateness Measure Set for LLMs in Asian Languages), a benchmark designed to evaluate entity-centric cultural biases across nine Asian languages representing six distinct cultures: Chinese, Japanese, Korean, Vietnamese, Urdu, Hindi, Malayalam, Marathi, and Gujarati.

Data Construction

The construction of Camellia involved a year-long collaboration with native speakers and consisted of two primary components:

1. Cultural Entities (19,530 total):

   • Asian-Centric Entities (15,821): Manually annotated entities specific to the six Asian cultures, categorized into six types: authors, food dishes, beverages, first names (male and female), locations, and sports clubs.
   • Western-Centric Entities (3,709): Parallel entities associated with North America and Europe, translated and adapted across all nine languages to ensure comparability.
   • Source: Entities were extracted from Wikidata and the mC4 web-crawl corpus, followed by manual filtering and annotation by native speakers to verify cultural relevance.

2. Masked Contexts (2,173 total):

   • Derived from natural social media discussions (X/Twitter, Weibo, Xiaohongshu).
   • Categorized into three types:
     • Camellia-Grounded: Contexts where only an entity from the specific Asian culture is appropriate (e.g., "The [MASK] dish is famous in Korea").
     • Camellia-Neutral: Contexts where entities from any culture are appropriate (e.g., "I am eating [MASK] today").
     • Camellia-QA: Paragraph-long contexts requiring implicit entity identification for extractive Question Answering.
   • Each context includes sentiment labels (positive, negative, neutral) and English translations.

Evaluation Framework

The benchmark evaluates four recent multilingual LLM families (Llama3.3-70b, Qwen2.5-72b, Aya-expanse-32b, and Gemma3-27b) across three tasks:

1. Cultural Context Adaptation: Measuring the Cultural Bias Score (CBS), which calculates the percentage of times a model assigns a higher likelihood to a Western entity over an Asian entity in a culturally grounded context.
2. Sentiment Association: Analyzing whether substituting Asian vs. Western entities in a sentence alters the model's predicted sentiment (measuring false negative/positive rates).
3. Entity Extractive QA: Evaluating the accuracy of extracting entities from long contexts, comparing performance on Asian vs. Western entities.

Key Results

1. Struggles with Cultural Adaptation

Models frequently fail to distinguish between Asian and Western entities in culturally grounded contexts.

• High CBS: Across most languages and models, the Cultural Bias Score (CBS) ranged between 30% and 40%. This indicates that in a significant portion of cases, models prefer Western entities even when the context explicitly requires an Asian one.
• Lack of Sensitivity: Performance did not significantly improve in culturally neutral contexts, suggesting models lack sensitivity to cultural grounding cues.
• Model Variance: Performance varied by model origin. For instance, Qwen2.5-72b (developed by Alibaba) performed significantly better on Chinese, Japanese, and Korean tasks compared to other models, likely due to better pre-training data coverage.

2. Divergent Sentiment Associations

Different model families exhibited distinct biases in how they linked cultures to sentiment:

• Llama and Gemma tended to associate Western entities with negative sentiment more frequently.
• Qwen often associated Asian entities with positive sentiment, particularly in Indian languages.
• These findings suggest that models developed in different regions may encode conflicting cultural biases, impacting downstream fairness.

3. Performance Gaps in Entity Extraction

• Asian Languages: In extractive QA tasks, models generally achieved higher accuracy when extracting entities associated with the respective Asian culture compared to Western entities. However, significant accuracy gaps (10–20%) were observed in several languages (e.g., Vietnamese, Urdu, and various Indian languages), indicating a struggle to grasp implicit cultural context in non-English settings.
• English Baseline: When tested on parallel English translations, these cultural gaps largely disappeared (mostly 1–5%), confirming that the bias is specific to the handling of non-English cultural contexts.

4. Tokenization Correlation

An analysis of model tokenizers revealed a correlation between script coverage in the tokenizer vocabulary and model performance. Models with higher coverage of a specific language's script (e.g., Qwen for CJK scripts, Llama for Indic scripts) tended to exhibit lower Cultural Bias Scores, suggesting that pre-training data representation is a key factor in mitigating bias.

Significance and Claims

The paper claims that Camellia provides the first comprehensive, entity-centric benchmark for measuring cultural biases in nine Asian languages. Its significance lies in:

• Revealing Systemic Bias: Demonstrating that current multilingual LLMs struggle to adapt to non-Western cultural contexts, often defaulting to Western associations even when inappropriate.
• Highlighting Regional Disparities: Showing that model performance and bias profiles are heavily influenced by the region of development and the composition of pre-training data.
• Providing a Resource: Offering a manually curated dataset of 19,530 entities and 2,173 masked contexts to support future research in developing more culturally fair and efficient multilingual LLMs.

The authors conclude that while LLMs are becoming more multilingual, they remain sensitive to cultural associations in ways that can lead to biased decision-making and poor performance in specific cultural domains. The benchmark serves as a tool to measure these disparities and guide the development of more equitable models.