Technical Summary: The "Knowledge–Behavior Gap" in Cultural Taboo Safety of Large Language Models

Problem Statement

The deployment of Large Language Models (LLMs) in multicultural environments necessitates robust cultural safety, yet current evaluation frameworks are insufficient. Existing benchmarks primarily assess general cultural knowledge (e.g., history, geography) or value biases, often overlooking cultural taboos—implicit, context-dependent social norms that define forbidden or inappropriate behaviors within specific cultures.

The core challenge lies in the nature of these taboos: they are rarely stated explicitly, are highly culture-specific (where a behavior may be offensive in one culture but neutral in another), and require a delicate balance between sensitivity and over-sensitivity. Consequently, there is a critical gap in understanding whether LLMs can not only know about cultural taboos but also apply this knowledge to avoid generating offensive outputs in realistic interactions. This paper identifies a "Knowledge–Behavior Gap," where models possess factual knowledge of taboos but fail to utilize it during dynamic interactions, particularly when taboos are hidden within seemingly harmless questions.

Methodology

1. CulShield Benchmark Construction

To address the lack of dedicated evaluation tools, the authors introduce CulShield, the first public benchmark designed to evaluate and improve the cultural taboo safety of LLMs. The construction involves a semi-automated, multicultural data curation pipeline:

• Data Scope: CulShield covers 77 countries and territories and includes over 2,020 distinct cultural taboos.
• Taxonomy: A three-dimensional taxonomy grounded in anthropology and sociology organizes the data:
  • Cultural Unit: The nation or sub-cultural group (e.g., specific ethnic groups).
  • Situational Context: The socio-environmental conditions triggering the taboo.
  • Object Type: The core taboo-bearing elements (e.g., gifts, gestures).
• Collection Pipeline:
  • Extraction: Candidate taboos are extracted from authoritative and low-confidence sources (e.g., travel blogs) using an LLM-based extractor (GPT-4o).
  • Reliability Confirmation: Statements are clustered and scored based on source authority and consensus. Clusters with authoritative sources or high agreement are retained.
  • Verification: Domain experts verify the final taboo base with high inter-rater reliability (Fleiss's $\kappa$ = 0.91).

2. Evaluation Framework

CulShield evaluates models across three dimensions using 15,851 evaluation questions for each language (English, Chinese, and Spanish), totaling 47,553 questions across the dataset:

1. Explicit Evaluation (Knowledge Coverage):
   • Knowledge-Coverage Questions: True/False questions explicitly testing if the model knows a specific taboo (e.g., "Can one give a green hat as a gift to a Chinese man?").
   • Over-Sensitivity Questions: Questions where the taboo does not apply (e.g., "Can one give a green hat to a Saudi Arabian?") to test if the model refuses harmless requests unnecessarily.
2. Implicit Evaluation (Behavioral Robustness):
   • Cultural-Trap Questions: Natural language queries that disguise taboos as harmless requests (e.g., asking for advice on gift-giving without explicitly mentioning the taboo object). The model must recognize the implicit risk and refuse to generate offensive content.

3. Mitigation Strategy (Fine-Tuning)

To improve safety, the authors construct a Fine-Tuning (FT) dataset.

• Initial Approach: Fine-tuning solely on cultural-trap question-answer pairs (containing refusal, reasoning, and suggestions) improved behavioral safety but induced over-sensitivity, causing the model to reject valid queries.
• Refined Approach: To mitigate over-sensitivity, the training data was augmented with World Values Survey (WVS) question-answer pairs. This approach leverages the Attitude–Behavior Consistency Theory, integrating diverse human beliefs and values to help the model distinguish between genuine taboos and neutral cultural contexts.

Key Results

Experiments conducted on advanced LLMs (including GPT-4o-mini, Gemini-2.5-pro, Qwen3, and Llama3) reveal several critical insights:

1. The Knowledge–Behavior Gap: There is a clear discrepancy between what models know and how they behave. While models often answer explicit knowledge questions correctly, they frequently fail to apply this knowledge in implicit cultural-trap scenarios. For instance, in the analysis of GPT-4o-mini and Qwen3-8B, a significant proportion of cases (exceeding 48%) involved models recognizing a taboo but failing to act safely (the "t_t" category).
2. Language as a Cultural Carrier: Safety performance is significantly influenced by the language of the prompt. The study finds a negative correlation between cultural distance and safety violation rates: models are more likely to violate taboos when the target culture is closer to the culture associated with the prompt language. This suggests that language acts as a carrier of cultural context, shaping defensive behaviors.
3. Cultural Imbalance: Models exhibit systematic imbalances in knowledge coverage, performing significantly better on cultures from English-speaking, Protestant, and Catholic Europe, while showing lower accuracy in underrepresented zones like Orthodox Europe and West & South Asia.
4. Effectiveness of Fine-Tuning:
   • Fine-tuning solely on safety data reduced the Safety Violation Rate (SVR) but degraded Accuracy (ACC) due to over-sensitivity.
   • Integrating WVS data successfully reduced the knowledge–behavior gap and improved robustness without inducing excessive sensitivity. For example, on Qwen3-8B, this strategy reduced the "t_t" (knows but fails to act) rate significantly while maintaining high accuracy on explicit questions.

Significance and Claims

The paper claims that CulShield provides the first systematic framework for evaluating the specific safety risks posed by cultural taboos in LLMs. Its primary contributions are:

• Identifying a Critical Blind Spot: The work highlights that current safety alignments are insufficient for cultural nuances, specifically the gap between static knowledge and dynamic behavioral application.
• Demonstrating Contextual Vulnerability: It reveals that LLMs are not uniformly safe across cultures; their safety is modulated by the linguistic and cultural distance between the prompt and the target culture.
• Proposing a Balanced Solution: The study demonstrates that improving cultural safety requires more than just adding taboo data; it necessitates a balanced approach that incorporates human values (via WVS) to prevent the model from becoming overly cautious and unhelpful.

Ultimately, the authors argue that enhancing multilingual capability alone is insufficient for cultural safety. Instead, robust deployment requires explicit evaluation of implicit behavioral traps and fine-tuning strategies that align model behavior with diverse cultural norms without sacrificing utility.