Characterizing Cultural Localization in AI-Generated Stories

Problem Statement
As Large Language Models (LLMs) are deployed globally, there is increasing interest in their ability to generate culturally localized content. However, it remains unclear whether models achieve localization through holistic localization—where cultural context shapes narrative plots, values, themes, and structures—or through templated localization. Templated localization occurs when a model inserts isolated cultural markers (e.g., names, locations, artifacts) into a culturally-agnostic, homogeneous narrative template. The paper posits that if models rely on templated localization, they risk producing stories that reflect homogeneous narratives and values, potentially leading to cultural harms such as erasure, the imposition of Western values, and reduced creative diversity. Furthermore, existing evaluations often fail to distinguish between surface-level lexical variation and deeper narrative differences, potentially overestimating a model's cultural competence.

Methodology
The authors propose a two-stage method to detect and measure the degree of templated localization in AI-generated stories:

1. Identifying Cultural Markers: The method first identifies the minimal set of lexical tokens (words) that distinguish stories across different nationalities. Using Normalized Pointwise Mutual Information (NPMI), the authors score words based on their association with specific nationalities. They then iteratively mask the top $k\%$ of these culturally associated words. The threshold $k$ is determined by finding the point where a classifier can no longer distinguish the nationality of the story better than random guessing. The remaining text, with these markers removed, is termed the "template image."
2. Measuring Narrative Homogeneity: The second stage measures the similarity of the "template images" across different nationalities using multi-word sequence metrics: Longest Common Subsequence (LCS) and Jaccard similarity of 4-grams. If the remaining narratives are highly similar across cultures after removing cultural markers, it indicates the presence of a shared, culturally-agnostic narrative template. This is compared against the similarity of the original stories and stories where random words were masked.
3. Characterizing Stereotypicality: Finally, the authors characterize the identified cultural markers for stereotypicality and offensiveness using the SeeGULL dataset. They calculate the precision of stereotypes within the markers and measure the average offensiveness score of these markers.

Experimental Setup
The study evaluated stories generated by five LLMs (GPT-3.5 Turbo, GPT-4o Mini, Llama 3.1 8B, Llama 3.3 70B, and Gemma 3 12B) across 193 nationalities and 125 story topics derived from cultural value frameworks (e.g., World Values Survey, Hofstede's Cultural Dimensions). The prompts instructed models to write children's stories in English about specific topics for a kid of a specific nationality.

Key Results

• Limited Vocabulary for Localization: The study found that cultural variation is concentrated in a very small subset of the vocabulary. Masking only 9% to 17% of culturally associated words was sufficient to make stories across different nationalities indistinguishable to a classifier.
• Presence of Shared Templates: After removing these cultural markers, the remaining narratives ("template images") exhibited significantly higher multi-word similarity across nationalities than the original stories. In many cases, the inter-group similarity of the masked stories accounted for over 70% of the intra-group similarity, suggesting a latent, shared narrative structure that is culturally agnostic.
• Stereotypicality and Offensiveness: While many cultural markers were neutral, the study identified that markers for 19 countries were, on average, offensive. These countries are predominantly located in the Global South (specifically Africa and West Asia) and often have lower-resource dominant languages. The offensive markers frequently aligned with negative stereotypes (e.g., "unreliable," "violent," "backward").

Significance and Claims
The paper claims that current AI-generated stories primarily exhibit templated localization rather than holistic localization. The authors argue that cultural representation in these models is reduced to a small number of recognizable symbols inserted into a homogeneous narrative worldview.

The significance of this work lies in its analytical lens for extrinsic cultural competence. The authors assert that evaluations focusing solely on the presence of culturally salient entities (like names or places) may overestimate a model's ability to generate localized narratives. By revealing the underlying homogeneity of the narrative structure, the paper highlights the need for:

1. Broader assessments of cultural competence that include narrative structures, values, stance, and dialect, not just surface markers.
2. Recognition that even without explicit template-based generation pipelines, LLMs may implicitly revert to globally dominant narrative schemas.
3. The development of methods to detect and measure narrative homogenization throughout the reasoning and tool-use processes of AI systems.

The authors conclude that understanding these mechanisms is crucial for anticipating and avoiding potential cultural harms, such as the erasure of diverse narrative traditions and the propagation of offensive stereotypes, particularly for communities in the Global South.