Lessons Without Borders? Evaluating Cultural Alignment of LLMs Using Multilingual Story Moral Generation

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities in generating text, their ability to understand and reproduce culturally specific narrative interpretations remains under-evaluated. Current benchmarks often rely on static knowledge tests or binary moral judgments (right/wrong), which fail to capture the nuanced, subjective process of how humans derive "story morals" (the central lesson or value inferred from a narrative).

The authors argue that story morals are not universal; they vary based on cultural context, linguistic framing, and individual reader expectations. The core research question is: Do frontier LLMs reproduce the patterns of interpretive variation observed in human readers across different languages and cultures, or do they flatten these differences into a homogenized, culturally neutral output?

2. Methodology

The authors introduce a novel evaluation framework centered on Multilingual Story Moral Generation. The methodology consists of four main components:

A. Dataset Construction

• Source Material: 14 novels from 14 distinct language-culture pairs (e.g., Arabic-Egypt, Korean-Korea, Portuguese-Brazil) were selected.
• Parallel Design: Plot summaries were extracted and translated into all 14 languages using DeepL and Google Translate, creating a fully crossed dataset (14 stories × 14 languages = 196 passages).
• Human Annotations: Using Prolific, the authors collected 588 human-written morals (3 per story-language pair). Annotators were filtered by native language fluency and geographic location to ensure cultural relevance.
• AI Contamination Mitigation: To address the risk of AI-generated human data, the authors flagged and manually removed ~12% of human morals that exhibited abnormally high semantic similarity to LLM outputs.

B. Model Evaluation

The study tested 7 models:

• Frontier Models: GPT-4o, Gemini 2.5 Flash.
• Open/Smaller Models: Gemma3 (4B/8B), Phi3 (8B), Aya (8B/35B), Qwen3 (8B).
• Prompting Strategy: Models were prompted to simulate a native speaker from the specific culture associated with the story language (socio-demographic prompting).

C. Three-Pronged Evaluation Framework

The authors employed three complementary methods to evaluate model performance against human baselines:

1. Semantic Similarity Analysis (Automated):

   • Used embedding models (LaBSE, paraphrase-multilingual-MiniLM, all-mpnet-base-v2) to calculate cosine similarity between moral pairs.
   • Hypotheses Tested:
     • H1: Does translation distort human baselines? (Result: No significant effect).
     • H2: Do human morals vary across cultures? (Result: Yes, though modestly; cross-lingual similarity is lower than intra-lingual).
     • H3: Do models fall within human intra-lingual variance? (Quality check).
     • H4: Do models exhibit comparable cross-lingual variance to humans? (Cultural sensitivity check).

2. Human Preference Survey:

   • Participants from specific cultural backgrounds were asked to choose their preferred moral from pairs generated under different conditions (In-Story/Out-Story vs. In-Culture/Out-Culture).
   • This tested whether humans prefer morals that are culturally specific and story-appropriate.

3. Value Categorization (Schwartz's Theory):

   • Used the MoVa framework to map generated morals to Schwartz's 10 Universal Values (e.g., Security, Benevolence, Power, Achievement).
   • This analyzed the distribution of underlying moral themes to see if models over-represent common values or miss rare, culturally specific ones.

3. Key Results

A. Semantic Similarity and "Cultural Flattening"

• High Human-Model Similarity: Frontier models (GPT-4o, Gemini) generated morals that were semantically very close to human responses, often exceeding the baseline agreement between two different humans (Human-Human similarity).
• Lack of Cross-Linguistic Variation: While humans showed distinct moral interpretations across different languages (cross-lingual similarity was lower), LLMs exhibited significantly higher cross-lingual similarity.
  • Finding: LLMs tend to generate the "same" moral regardless of the language or cultural context. For example, the concept of "Justice" appeared in 12/14 GPT-4o morals for a single story across all languages, whereas human responses were more diverse.
  • Conclusion: Models approximate the central tendency of human moral interpretation but fail to reproduce the diversity and cultural specificity that characterizes human understanding.

B. Human Preference Survey

• Story Specificity: Humans strongly preferred morals that were specific to the story content rather than universal platitudes.
• Cultural Preference: Humans showed a slight but measurable preference for morals generated by people from their own cultural background.
• The "Uncanny Valley" of Preference: Surprisingly, LLM-generated morals were consistently preferred by human participants over human-generated morals, even when the LLM output was not culturally specific. This suggests a human bias toward the polished, "unmarked" style of LLM outputs.

C. Value Distribution (Schwartz Analysis)

• Over-representation of Common Values: LLMs disproportionately emphasized widely shared values like Security, Benevolence, and Self-Direction.
• Under-representation of Rare Values: Models significantly under-represented values like Power, Achievement, and Hedonism, which appeared more frequently in human-generated morals.
• Implication: Models appear to favor "prototypical" and socially safe moral themes, effectively smoothing out the controversial or culturally specific nuances found in human data.

4. Key Contributions

1. Novel Evaluation Task: Introduces Multilingual Story Moral Generation as a culturally grounded benchmark, moving beyond static knowledge tests to evaluate dynamic narrative interpretation.
2. Methodological Framework: Proposes a three-part evaluation strategy (Semantic Similarity, Human Preference, Value Categorization) to holistically assess cultural alignment.
3. Empirical Evidence of Cultural Flattening: Provides quantitative evidence that while frontier models can mimic human moral reasoning, they systematically reduce cross-cultural variance, producing a homogenized "global" moral perspective rather than culturally situated interpretations.
4. Dataset Release: Publicly released a parallel dataset of 14 stories across 14 languages with human and model-generated morals.

5. Significance and Implications

• Limitations of Current Alignment: The findings suggest that current "alignment" techniques may be optimizing for a global average of human values, inadvertently erasing cultural diversity. This poses a risk for applications requiring cultural sensitivity (e.g., education, mental health, journalism).
• Beyond "Correctness": The paper argues that narrative understanding cannot be evaluated by finding a single "correct" moral. Instead, evaluation must account for the distribution of interpretations across different communities.
• Future Directions: The authors call for future work to develop benchmarks that explicitly reward cultural variation and to investigate how to prevent models from collapsing diverse human perspectives into a single, flattened output.

In summary, the paper demonstrates that while LLMs are becoming better at mimicking human moral reasoning, they are currently failing to reproduce the rich, culturally specific diversity of human narrative understanding.