Technical Summary: CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries

Problem Statement

Large Language Models (LLMs) currently function as global information intermediaries but frequently default to "WEIRD" (Western, Educated, Industrialized, Rich, Democratic) value systems, leading to persistent cultural bias. A critical gap exists in evaluating cultural competency—the ability of AI to infer and adapt to a user's implicitly signaled cultural values rather than relying on static demographic traits.

Current benchmarks largely treat cultural identity as a binary attribute (belonging vs. not belonging), ignoring the fluidity and intra-cultural variation of real human identity. This reductive approach risks misalignment and the reinforcement of harmful stereotypes, particularly in high-stakes domains like healthcare where communication style directly impacts trust and health outcomes. Furthermore, users rarely declare their heritage explicitly; instead, they signal context through implicit narratives, communication preferences, and social cues. Existing models lack the sensitivity to recognize these subtle signals.

Methodology: The CCBENCH Framework

The authors introduce CCBENCH, a domain-agnostic framework designed to evaluate the implicit cultural competence of LLMs. Unlike traditional benchmarks relying on explicit self-identification (e.g., "As a person from culture X..."), CCBENCH measures a model's ability to infer latent cultural norms through behavioral cues embedded in conversation history.

1. Framework Design

The framework operationalizes cultural identity as a continuum of norm adherence states revealed implicitly. It utilizes a hierarchical representation distinguishing between abstract cultural values and the behavioral norms that operationalize them.

• Persona Construction: Synthetic personas ($P_k$) are generated based on a set of core cultural values ($V$) and behavioral norms ($N$).
• Adherence Logic:
  • Value Adherence: Each persona is assigned a binary adherence state ($+1$ for adherence, $-1$ for non-adherence) to specific cultural values.
  • Norm Derivation: A persona's adherence to a specific norm is calculated as the mean of its associated value scores.
  • Ternary States: The final norm adherence state ($C_k$) is categorized into three states: Follow ($\mu > 0$), Avoid ($\mu < 0$), or Neutral ($\mu = 0$). This allows for nuanced personas that may follow some norms while avoiding others.

2. CCBENCH-Health Benchmark Creation

The framework is instantiated as CCBENCH-Health, focusing on health-related queries.

• Data Sourcing: Norms and values are grounded in the Mosaica resource, a knowledge base of cross-cultural insights curated by experts.
• Persona Simulation: 60 theoretically grounded personas were created across six cultures (Afghan, Burmese, Chinese, Maori, Nepali, and Vietnamese). Each persona engages in 18 realistic dialogues.
• Implicit Grounding: A multi-turn conversation history ($H_k$) is generated between the persona and an AI assistant. This history is constrained by the adherence function $C_k$ such that the persona demonstrates values through linguistic style and decision-making patterns without explicitly stating their geographic or cultural background. Explicit identity statements are filtered out, while cultural artifacts (e.g., greetings, religious references) are retained as implicit cues.
• Query Curation: 52 authentic healthcare questions were drawn from real user forums (eHealth, iCliniq), stratified across 12 health topics.
• Evaluation Protocol:
  • Checklist-Based Evaluation: For each interaction, a norm-specific checklist is generated based on the persona's adherence state. A strong evaluator LLM scores the model's response ($R_i^k$) on the proportion of satisfied checklist items.
  • Metrics:
    • Follow Rate: Frequency of adapting to norms a persona prefers.
    • Avoid Rate: Frequency of correctly adapting to the avoidance of cultural norms (resistance to stereotyping).
    • Cultural Competency Score (CCS): The harmonic mean of Follow and Avoid rates.

3. Experimental Setup

Five leading LLMs (GPT-5.2, Gemini-2.5-Pro, Claude-3.5-Sonnet, Llama-3.2-90B, Qwen-3.5-397B) were evaluated under four prompting configurations:

1. No Context: Only the health query.
2. With Conversation Context (Hist.): The simulated background history is provided.
3. Culture-CoT: The model is instructed to perform Chain-of-Thought reasoning to identify cultural cues before answering.
4. Norm Context: The model is explicitly provided with the ground-truth norm definitions (theoretical upper bound).

Key Results

Benchmarking reveals a pervasive deficit in adapting to implicit norm reveals, characterized by a significant asymmetry between "Avoid" and "Follow" capabilities.

• Limited Overall Competence: Even the best-performing models achieve culturally appropriate responses only 20–30% of the time (measured by CCS) when explicitly prompted with norm states. In implicit settings (Conversation Context), performance drops significantly.
• The Asymmetry (Western Default): Models perform significantly better when personas prefer to avoid cultural norms than when they prefer to follow them.
  • Example: GPT-5.2 achieves a 51.7% Avoid Rate but only a 6.5% Follow Rate under conversational context.
  • This suggests models possess a "Western default" where the omission of non-Western norms is misinterpreted as competence, while active cultural accommodation remains weak.
• Impact of Explicit Cues: Explicitly prompting models to focus on cultural cues (Culture-CoT) yields modest improvements (3–5% average gain), indicating that reasoning scaffolds help but cannot substitute for targeted cultural adaptation.
• Cultural Disparities:
  • Afghan Context: Models struggle most here, with an average Cultural Competence Score of 0.088. Cultural cues rarely yield appropriate health advice.
  • Maori Context: Exhibits notably high Avoid Rates, likely due to distinctive linguistic markers that are easier for models to detect and suppress.
• Style vs. Substance: Models sometimes adapt more readily to implicit communication styles (e.g., hesitation) than to explicit cultural practices, though this varies by culture.
• Norm Adherence Correlation: As a persona's cultural embeddedness (adherence to norms) increases, model adaptation declines sharply. Models treat high cultural specificity as outlier behavior, defaulting to generalized responses.

Significance and Claims

The paper claims that CCBENCH provides a rigorous method for evaluating how LLMs handle the fluidity and multiplicity of human values, moving beyond binary cultural belonging.

The primary significance of the work lies in exposing a "competency gap":

1. Resistance vs. Sensitivity: Modern LLMs are proficient at resisting stereotyping (correctly identifying when a user does not adhere to a norm) but consistently struggle with cultural sensitivity (actively integrating non-Western values when signaled).
2. Insufficiency of Current Alignment: Even when provided with explicit norm definitions as a theoretical upper bound, models fail to consistently adapt, suggesting current alignment techniques are insufficient for true cultural competence.
3. Call for Nuanced Care: The findings argue against "one-size-fits-all" recommendations. The authors assert that achieving equitable cultural competence requires richer multicultural training data and techniques that respect the diverse, dynamic, and often implicit nature of global identities.

The authors conclude that while the benchmark is limited to six cultures and relies on simulated personas, it serves as a necessary foundation for developing AI that can navigate intersectional, multifaceted cultural contexts in high-stakes domains like healthcare.