💬 NLP

Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectations

This paper introduces a human-centered framework to evaluate LLM cultural alignment by comparing model-generated representation vectors against human-derived importance vectors across nine countries, revealing that current frontier models exhibit Western-centric biases and systemic errors that fail to authentically capture the nuanced cultural priorities of non-US populations.

Original authors: Erin MacMurray van Liemt, Aida Davani, Sinchana Kumbale, Neha Dixit, Sunipa Dev

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Erin MacMurray van Liemt, Aida Davani, Sinchana Kumbale, Neha Dixit, Sunipa Dev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a tour guide for a massive, digital library that contains the stories of every country on Earth. This library is run by a super-smart robot (an AI) that has read billions of books, websites, and articles.

The paper you're asking about asks a very simple, yet profound question: If you ask this robot to describe your country's culture, does it tell the story the way you would tell it, or does it tell the story the way a tourist from America would tell it?

Here is the breakdown of the research, using some everyday analogies.

1. The Problem: The "Tourist Gaze" vs. The "Local's Heart"

For a long time, scientists checked if AI was "culturally smart" by asking: Does the robot know the facts?

The Old Way: "Does the robot know that Japan has sushi?" (Yes/No).
The New Way (This Paper): "When you think about Japan, what comes to mind first? Is it the sushi, or is it the quiet respect people show each other? Is it the temples, or is it the way families gather?"

The researchers call this "Cultural Authenticity." It's the difference between a postcard (which shows the pretty, obvious stuff) and a diary (which shows the real, deep, daily life).

The Analogy:
Imagine you ask a stranger to describe your hometown.

The "Tourist" Description: They talk about the famous statue in the square, the local pizza, and the big stadium. They hit all the "checklist" items.
The "Local" Description: You talk about the specific way your neighbors wave, the smell of the bakery on Tuesday mornings, the unspoken rules about how to behave at the park, and the values your family holds dear.

The paper found that current AI models (like Gemini, GPT-4, and Claude) are acting like over-enthusiastic tourists. They list everything on the "checklist" but miss the deep, emotional priorities that actual locals care about.

2. How They Tested It: The "Two-Party" Experiment

The researchers ran two separate studies to compare the "Local" view with the "Robot" view.

Study 1: The Human Baseline (The "Local" View)
They went to 9 different countries (like India, Brazil, France, Japan, etc.) and asked real people a simple question: "When you think about your culture, what is the most important thing?"

People wrote down their answers freely.
The researchers turned these answers into a "Cultural Priority Map."
Example: For people in India, "Social Customs" and "Religious Rituals" were huge priorities. For people in France, "Food" and "Language" were top of the list.

Study 2: The Robot's View (The "Tourist" View)
Then, they asked three of the smartest AI robots to describe those same 9 countries. They asked the robots the same questions, but in many different ways (like asking for a story, a list, or a description) to see if the robots would change their minds.

The researchers turned the robot's answers into a "Cultural Representation Map."

3. The Big Discovery: The "Western Filter"

When they compared the two maps, they found some surprising patterns:

The "US Proximity" Rule: The closer a country is culturally to the United States, the better the AI understood it.
- Analogy: If you ask the robot about France or Italy, it gets it mostly right. But if you ask about India or Indonesia, the robot starts to drift. It prioritizes "exotic" things (like religious rituals or traditional clothing) that a tourist would notice, while ignoring the deep social values that locals actually care about.
The "Encyclopedia" Mistake: The robots tried to be too helpful. Instead of highlighting what locals thought was most important, the robots tried to list everything equally.
- Analogy: Imagine a local saying, "My culture is about how we treat our elders." The robot replies, "Okay, here is a list: We treat elders, we eat rice, we wear kimonos, we have temples, we dance, we have sports..."
- The robot gave a flat list (an encyclopedia) instead of a hierarchy (a story with a beginning, middle, and end). It missed the weight of what matters.

4. The Scary Part: All Robots Make the Same Mistake

The researchers found that the three different AI models (Gemini, GPT, and Claude) all made almost the exact same mistakes.

The Correlation: Their errors were 97% identical.
The Meaning: This means the problem isn't that one specific robot is "dumb." The problem is that all of them were trained on the same internet data.
The "Echo Chamber": The internet is full of content written by Westerners about the rest of the world. So, the robots are learning a "Western Tourist's View" of the world, not the "Local's View." They are all echoing the same bias.

5. Why This Matters

The paper argues that we need to stop just checking if AI is "factually correct" (e.g., "Is the capital of Peru Lima?") and start checking if AI is "culturally authentic" (e.g., "Does the AI understand what Peruvians value most?").

If we don't fix this, AI will continue to act like a digital colonialist:

It will keep defining cultures from the outside in.
It will keep erasing the "backstage" reality of how people actually live and think.
It will keep prioritizing the "exotic" over the "essential."

The Takeaway

The researchers are saying: "We need to teach AI to listen to the locals, not just read the travel brochures."

They propose a new way to test AI: Don't just ask, "Do you know our culture?" Ask, "Do you know what we think is important about our culture?" If the AI can't answer that, it's not truly aligned with the people it's supposed to serve.

1. Problem Statement

Current evaluations of Large Language Models (LLMs) regarding cultural representation primarily rely on cultural diversity (variety of outputs) and factual accuracy. However, a critical gap exists in assessing cultural alignment: the degree to which generated content mirrors how native populations actually perceive, prioritize, and define their own cultures.

Existing methods often:

Rely on top-down, pre-defined taxonomies (e.g., Hofstede's dimensions) that may not reflect lived realities.
Assume LLMs possess an underlying "system of values" comparable to humans, which is increasingly questioned.
Fail to account for representational agency: the power of a community to define its own identity, rather than having it defined by external (often Western) lenses.

The paper argues that true cultural alignment requires shifting the evaluation goal from factual accuracy to Cultural Authenticity, measuring whether a model's information hierarchy reflects the priorities of a country's residents.

2. Methodology

The authors propose a human-centered framework using a dual-study design across nine countries (Brazil, France, Germany, India, Indonesia, Italy, Japan, Mexico, South Korea) to compare human expectations against model observations.

Study 1: Establishing the Human-Centric Baseline (Expectation)

Data Source: Open-ended survey responses from 5,629 participants across 13 countries (focusing on 9 for this study). The prompt asked: "When you think about your culture, what kinds of cultural artifacts, practices or other attributes do you consider most important?"
Facet Induction: Responses were mapped to a structured taxonomy of 11 Cultural Facets:
1. Architecture/Physical Spaces
2. Performance and Art
3. Fashion/Style
4. Events
5. Cuisine
6. Important Figures
7. Communication
8. Religious Rituals
9. Social Practices/Customs
10. Sports
11. Values, Norms, Beliefs, and Morality (VNBM)
Labeling: An automated pipeline (Gemini 2.5 Flash) classified responses into these 11 facets, validated by human review (97% precision).
Output: Cultural Importance Vectors ( $v_c$ ) for each country, representing the normalized distribution of facet priorities derived from human responses.

Study 2: Measuring Model Representations (Observation)

Models Evaluated: Three frontier LLMs: Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Haiku.
Prompting Strategy: To avoid bias from specific phrasing, the authors used syntactic diversification. They generated 85 distinct prompt templates per country, varying sentence structure (active, passive, imperative, etc.) and use cases (information seeking vs. content generation).
Sampling: Each template was sampled 5 times at varying temperatures ( $T \in \{0, 0.5, 1\}$ ), resulting in 425 responses per model per country.
Analysis: An autorater (Gemini 2.5 Pro) identified the presence of the 11 cultural facets in the model outputs.
Output: Cultural Representation Vectors ( $\hat{v}_{c,M}$ ) for each model, representing the normalized distribution of facets mentioned in the generated text.

Evaluation Metrics

The alignment between Human Expectation ( $v_c$ ) and Model Representation ( $\hat{v}_{c,M}$ ) was quantified using:

Pearson Correlation ( $\rho$ ): Linear relationship between model prioritization and user values.
Cosine Similarity ( $S_C$ ): Directional similarity in n-dimensional space (preservation of relative hierarchy).
Mean Squared Error (MSE): Penalizes large discrepancies in critical cultural pillars.

3. Key Contributions

Framework for Cultural Authenticity: Introduces a bottom-up, human-derived baseline to evaluate LLMs, shifting focus from "what is factually true" to "what is culturally prioritized."
Cultural Importance Vectors: Establishes a ground-truth dataset of cultural facet priorities for nine diverse nations, moving beyond WEIRD (Western, Educated, Industrialized, Rich, Democratic) biases.
Syntactic Diversification: A novel prompting technique to ensure model outputs are not artifacts of specific prompt phrasing but reflect genuine cultural knowledge retrieval.
Systemic Error Identification: Demonstrates that cultural misalignment is not random but a systemic, shared failure across different model architectures.

4. Key Results

RQ1: Western Proximity Impacts Order of Importance

Finding: There is a clear negative correlation between Cultural Distance from the US (measured by Cultural Fixation Index) and alignment.
Observation: As a country becomes culturally more distant from the US, model alignment with human expectations decreases.
Model Performance:
- GPT-4o: Best alignment for France, Italy, and Mexico.
- Gemini 2.5 Pro: Best alignment for the remaining six countries.
- Claude 3.5 Haiku: Consistently showed the lowest alignment scores.
- Gemini showed relatively stable performance across regions, while GPT and Claude showed significant degradation for non-Western cultures.

RQ2: Systemic Facet Misalignment

Finding: Models exhibit a "flattened" or "encyclopedic" view of culture.
Observation: LLMs tend to over-index equally on all cultural dimensions, failing to replicate the specific hierarchies of human priorities.
Specifics: Models often neglect deep-seated social values (VNBM) and specific local customs in favor of surface-level "touristic" markers (e.g., Architecture, Cuisine, Fashion) that are easily retrievable from training data.

RQ3: Systemic Error in Algorithmic Authenticity

Finding: There is an extraordinary convergence in error patterns across all three models.
Data: The error vectors (difference between model output and human baseline) showed near-total inter-model correlation:
- Gemini vs. GPT: $\rho = 0.971$
- Gemini vs. Claude: $\rho = 0.977$
- Claude vs. GPT: $\rho = 0.968$
Implication: The misalignment is not a model-specific bug but a systemic bias embedded in the underlying training paradigms and data corpora (likely a homogenized "digital mediascape" dominated by Western perspectives).

5. Significance and Discussion

The "Tourist Gaze": The paper draws on anthropological theory (John Urry) to describe LLM outputs as a "Tourist Gaze." Models present culture as a flattened list of exotic or consumable markers (Front Stage) rather than capturing the internal, nuanced priorities and suppressed facts of the culture (Back Stage).
Representational Agency: The results highlight a crisis of agency where the Global South's self-definition is overwritten by external, algorithmic definitions. Switching between frontier models offers no solution, as they share the same error signatures.
Future Directions: The authors advocate for moving beyond simple diversity metrics. Future work must focus on fidelity to human hierarchies and developing evaluation frameworks that respect the "representational agency" of diverse populations.

In conclusion, the paper provides empirical evidence that current frontier LLMs fail to authentically represent global cultures, not because they lack knowledge, but because they lack the prioritization logic that native humans use to define their own identities.