💬 NLP

TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories

This paper introduces TALES, a comprehensive framework comprising a taxonomy of cultural misrepresentations and a large-scale evaluation of 6 LLMs, which reveals that 88% of AI-generated stories about Indian cultures contain errors, particularly affecting mid- and low-resourced languages and peri-urban narratives.

Original authors: Kirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha, Shachi Dave, Danish Pruthi

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha, Shachi Dave, Danish Pruthi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot friend who loves to tell stories. You ask it to write a tale about a wedding in a small village in India, or a childhood memory in a bustling city. You expect the robot to capture the local flavor—the specific snacks, the right family titles, the unique traditions.

But according to this paper, TALES, that robot friend is often getting the details wrong. In fact, it's like asking a tourist who has only read a travel brochure to describe your hometown; they might get the big landmarks right, but they'll likely mix up the local slang, the food, and the daily routines.

Here is a simple breakdown of what the researchers found, using some everyday analogies:

1. The Problem: The Robot's "Travel Brochure" Brain

The researchers wanted to know: When AI tells stories about Indian culture, does it get it right?
They found that 88% of the stories generated by popular AI models contained mistakes. These weren't just tiny typos; they were cultural blunders.

Think of it like a chef trying to cook a regional dish.

The Mistake: The chef puts a dessert ingredient (like sugar) into a savory dish (like a curry) because they think "Indian food" is all one thing.
The Reality: The researchers found the AI often confused specific regional foods, used the wrong family titles (calling a male relative "aunt"), or described events that simply don't happen in real life (like a school bus having a ticket conductor).

2. The Tool: The "Cultural Cheat Sheet" (TALES-Tax)

Before they could count the mistakes, the researchers needed a way to categorize them. They didn't just guess; they asked real people from India (9 focus groups and 15 individual surveys) to read the AI stories and point out what felt "off."

Based on this feedback, they created a Cultural Cheat Sheet (called TALES-Tax) with seven categories of errors:

Cultural Inaccuracy: Getting facts wrong (e.g., saying a specific snack is eaten for breakfast when it's actually a snack).
Unlikely Scenarios: Making up events that feel fake (e.g., a street performer playing a complex classical instrument during a morning school assembly).
Clichés: Relying on stereotypes (e.g., assuming everyone in a region drinks a specific type of coffee or wears traditional festival clothes to school).
Oversimplification: Smoothing over unique details (e.g., calling all Indian art "Rangoli" instead of using the specific local name).
Factual Errors: Getting geography or history wrong (e.g., saying there is a desert next to a city that is actually green).
Linguistic Errors: Spelling names wrong or using family words incorrectly.
Logical Errors: Breaking the rules of how things work (e.g., a character bathing in a drinking water well).

3. The Big Test: The "Taste Test" with 108 Experts

The researchers didn't just look at one story. They hired 108 experts from 71 different regions across India. These were native speakers who knew their local cultures inside and out.

They asked these experts to read 540 stories written by 6 different AI models (including big names like GPT-4 and Gemini) in 14 different languages.

The Results:

The "Rich" vs. "Poor" Gap: The AI was much better at telling stories about big, famous cities (Tier-1) and in English. When the story was set in a smaller town or written in a less common Indian language, the mistakes quadrupled.
The "Generic" Trap: Some AI models tried to avoid mistakes by writing very boring, generic stories with no cultural details at all. Others tried to be creative but got the details wrong. Neither approach was good.
Relatability: Because of these mistakes, the stories felt "unrelatable" to the people they were supposed to represent. It's like watching a movie about your hometown where the actors speak with a fake accent and eat the wrong food; you just can't connect with it.

4. The Big Surprise: The "Knowledge vs. Performance" Paradox

Here is the most interesting part. The researchers wondered: "Is the AI just ignorant? Does it not know the facts?"

To test this, they turned the mistakes into a quiz (called TALES-QA). They asked the AI direct questions like, "What is the traditional food served at this wedding?" or "Where is this statue located?"

The Shocking Result:

When asked as a quiz, the AI got the answers right 76% of the time (in English) and 60% of the time (in Indian languages).
The Conclusion: The AI knows the facts. It has the information in its brain. But when it tries to write a story, it fails to use that knowledge correctly.

The Analogy:
Imagine a student who can ace a multiple-choice test on history (getting 90% right) but, when asked to write a history essay, makes up wild, incorrect facts. The student isn't "ignorant"; they just can't apply what they know in a creative, open-ended situation.

Summary

The paper concludes that current AI models are like overconfident tourists. They have read the guidebooks (they have the data), but when they try to tell a story about a local culture, they rely on stereotypes, mix up details, and fail to capture the true "flavor" of the place. This is especially true for smaller towns and less common languages.

The researchers hope their "Cultural Cheat Sheet" and "Quiz" will help developers fix this, so that future AI can tell stories that feel real and respectful to the people they are about.

Technical Summary: TALES – A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories

Problem Statement

While Large Language Models (LLMs) have advanced significantly in creative writing tasks, they frequently fail to accurately represent diverse non-Western cultures, often perpetuating stereotypes, factual errors, and cultural homogenization. Existing evaluations of cultural competence in AI largely focus on knowledge retrieval, value alignment, or adherence to norms in closed-ended tasks. There is a critical gap in evaluating cultural representation in open-ended, generative tasks (such as storytelling), where "correctness" is subjective and rooted in community lived experiences. Specifically, there is a lack of structured frameworks to identify how LLMs misrepresent cultural identities in India, a region characterized by immense linguistic, geographic, and cultural diversity.

Methodology

The authors propose TALES (Taxonomy and Analysis of Cultural Representations in LLM-generated Stories), a community-centered evaluation framework consisting of three main phases:

1. Developing a Taxonomy (TALES-Tax)

To address RQ1 (How do LLMs misrepresent diverse cultural identities?), the researchers conducted a qualitative study involving:

Focus Groups (N=9) and Individual Surveys (N=15) with participants from diverse Indian regions.
Participants reviewed LLM-generated stories (using GPT-4) based on topics like festivals, weddings, and daily life.
Using reflexive thematic analysis on participant comments and highlighted text spans, the team derived TALES-Tax, a taxonomy of seven categories of cultural misrepresentation:
1. Cultural Inaccuracy: Incorrect descriptions of symbols, traditions, or beliefs.
2. Unlikely Scenarios: Contextually implausible events based on local norms.
3. Clichés: Overuse of stereotypes or romanticized tropes.
4. Oversimplification: Homogenizing complex cultural elements.
5. Factual Error: Incorrect objective information (e.g., geography, history).
6. Linguistic Inaccuracy: Spelling, grammar, or inappropriate code-switching.
7. Logical Error: Narrative inconsistencies.

2. Large-Scale Human Evaluation

To address RQ2 (How frequent are these misrepresentations?), the team conducted a large-scale annotation study:

Models Evaluated: 6 popular LLMs (2 closed-source: GPT-4.1, Gemini 2.5 Pro; 4 open-source: Llama 3.3 70B, Gemma 3 27B, Qwen 3 32B, Aya 32B).
Annotators: 108 "expert" annotators with native language proficiency and lived experience from 71 regions across India, covering 14 languages (English + 13 Indic languages).
Data: 540 stories generated using prompts based on Hofstede's cultural onion model (Symbols, Heroes, Rituals, Values).
Annotation: Annotators highlighted spans of misrepresentation using TALES-Tax and rated story relatability on a 5-point Likert scale.
Analysis: Statistical analysis (Mann–Whitney U tests) compared misrepresentation frequencies across language resource levels (High, Mid, Low) and regional tiers (Tier-1, Tier-2, Tier-3).

3. Measuring Cultural Knowledge (TALES-QA)

To address RQ3 (Are misrepresentations due to a lack of cultural knowledge?), the researchers:

Converted the annotated misrepresentation spans into standalone questions (TALES-QA).
Used GPT-4.1 to generate candidate questions, which were then verified and refined by human experts.
Created a dataset of 1,683 questions (1,115 in Indic languages, 568 in English) covering various formats (One-word, Fill-in-the-blank, MCQ, True/False).
Evaluated the same 6 models on this question bank to determine if they possessed the underlying knowledge that they failed to apply in storytelling.

Key Results

Prevalence and Distribution of Misrepresentations

High Frequency: 88% of the generated stories contained at least one cultural misrepresentation. On average, there were 5.42 misrepresentations per story (roughly one every 5 sentences).
Language Disparity: Misrepresentations increased significantly in non-English languages.
- Mid-resource languages saw a 56% increase in misrepresentations compared to English.
- Low-resource languages saw a quadrupling of misrepresentations.
- Linguistic errors were the dominant category in non-English stories.
Regional Disparity: Stories set in Tier-3 (peri-urban/rural) regions contained significantly more misrepresentations than those in Tier-1 regions. Cultural inaccuracies and factual errors increased most sharply for Tier-3 regions.
Relatability: The average relatability rating was 3.25/5. There was a moderate negative correlation ( $r_s = -0.27$ ) between the number of misrepresentations and story relatability. Closed-source models generally received higher relatability scores than open-source models.

Cultural Knowledge vs. Generative Application

Knowledge Gap: Despite the high rate of misrepresentation in stories, models demonstrated surprisingly high accuracy on TALES-QA (standalone questions).
- English: Average accuracy was 76.9%.
- Indic Languages: Average accuracy dropped to 59.8%, with the lowest performance in low-resource languages.
Implication: The primary cause of misrepresentation is not a lack of cultural knowledge. Instead, models struggle to faithfully apply stored cultural knowledge when generating open-ended, context-rich narratives.

Model-Specific Findings

Gemini 2.5 Pro performed best overall, generating the most culturally rich stories (high Cultural Specific Item count) with the lowest fraction of misrepresentations.
LLaMA 3.3 70B generated fewer misrepresentations per story but also exhibited the lowest cultural richness (fewest cultural items), suggesting a tendency toward "no-representation" (generic text) rather than active misrepresentation.
Aya 32B and Qwen 3 32B generated high cultural richness but also high rates of misrepresentation.

Key Contributions

TALES-Tax: A novel, community-derived taxonomy of seven categories of cultural misrepresentations, developed through focus groups and surveys with Indian participants.
Large-Scale Evaluation: A comprehensive human evaluation involving 108 native speakers from 71 regions, providing the first granular analysis of cultural misrepresentation across 6 models, 14 languages, and diverse Indian regions.
TALES-QA: A standalone question bank of 1,683 questions designed to decouple cultural knowledge from generative capability, revealing that models often possess the knowledge but fail to apply it correctly in storytelling.
Empirical Evidence of Gaps: The study provides quantitative evidence that cultural misrepresentations are exacerbated in mid/low-resource languages and peri-urban regions, highlighting specific areas for improvement in AI development.

Significance and Claims

The paper claims that its work is significant because it moves beyond simple knowledge retrieval metrics to evaluate cultural representation in generative settings through the lens of community members' lived experiences.

Community-Centered Approach: By grounding the taxonomy and evaluation in the perspectives of diverse Indian participants, the study challenges the "Western-centric" design of current AI systems and highlights the specific harms faced by minoritized communities within minoritized regions (e.g., rural India).
Knowledge vs. Application: The finding that models can answer cultural knowledge questions correctly yet fail to represent culture accurately in stories suggests a fundamental gap in cultural alignment during generation. This indicates that simply increasing training data or knowledge bases is insufficient; models need better mechanisms to apply cultural context in open-ended tasks.
Future Directions: The authors argue that AI systems must be designed to handle cultural uncertainty (e.g., by abstaining or asking follow-up questions) rather than hallucinating cultural details. The release of TALES-Tax and TALES-QA provides a structured foundation for future research to mitigate these misrepresentations and improve cultural competence in non-Western contexts.

The authors maintain a modest stance, acknowledging limitations such as the use of English for initial focus groups and the specific prompt design, noting that their work serves as a starting point for understanding and evaluating cultural misrepresentations rather than a definitive endpoint.