🤖 AI

CULTURESCORE: Evaluating Cultural Faithfulness in Video Generation Models

This paper introduces CultureScore, a novel evaluation framework that decomposes cultural faithfulness into identity, context, and behavior dimensions to reveal that current state-of-the-art video generation models significantly struggle with culturally accurate representation, often prioritizing visual quality over cultural correctness.

Original authors: Anku Rani, Wei Dai, Shravan Nayak, Pattie Maes, Mahdi M. Kalayeh, Paul Pu Liang

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anku Rani, Wei Dai, Shravan Nayak, Pattie Maes, Mahdi M. Kalayeh, Paul Pu Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical movie camera that can create any scene you describe. You tell it, "Show me a family in India celebrating a festival," and it spits out a beautiful, high-definition video. But here's the catch: the family in the video is wearing Western suits, eating at a fancy dining table, and shaking hands instead of bowing with folded hands.

To a standard quality checker, this video might get a perfect score because the lighting is great, the people look real, and the camera moves smoothly. But to a local Indian person, the video feels wrong, like a costume party where everyone got the costumes mixed up.

This paper introduces a new way to test these "magic movie cameras" (AI video generators) called CULTURESCORE. It argues that just because a video looks pretty doesn't mean it looks true to a specific culture.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Perfectly Wrong" Video

Current tools that grade AI videos (like VideoScore) are like art critics who only look at the frame. They check: "Is the picture blurry? Is the color bright? Does the person move smoothly?"

The Flaw: If the AI replaces a traditional Indian greeting (a Namaste) with a Western handshake, the art critic gives it a high score because the handshake is drawn perfectly.
The Reality: For someone from that culture, the video is a failure. It's like serving a delicious-looking cake that tastes like soap.

2. The Solution: The "Three-Layer Cake" (CULTURESCORE)

The authors propose a new grading system called CULTURESCORE. Instead of just looking at the whole picture, they slice the video into three specific layers to see where the AI messed up:

Layer 1: Identity (Who is in the room?)
- The Analogy: Are the actors wearing the right clothes? Do they look like the people they are supposed to be?
- Example: If the prompt says "Japanese colleagues," are they dressed in typical Japanese business attire, or do they look like generic Western office workers?
Layer 2: Behavior (What are they doing?)
- The Analogy: Are they moving with the right rhythm?
- Example: A Namaste isn't just hands together; it's a specific bow and timing. A handshake is a different rhythm. The AI often gets the "dance steps" wrong, even if the actors look okay.
Layer 3: Context (Where are they?)
- The Analogy: Is the background set up correctly?
- Example: If it's a family dinner in a specific culture, are they sitting on the floor around a low table (a dastarkhwan) or at a tall Western dining table?

3. The Big Discovery: The "Inverted World"

The researchers tested three of the smartest video AI models (Veo, LTX-2, and Wan) with prompts from 10 different countries. They found a shocking pattern:

The "Hollywood" Model: One model (LTX-2) made the most "cinematic," high-quality videos. It got the highest scores on traditional quality checks.
The "Cultural" Model: Another model (Wan 2.2) made videos that were less "pretty" but much more culturally accurate.
The Twist: When real humans from those countries watched the videos, they hated the "Hollywood" model and loved the "Cultural" model.
- The Metaphor: It's like a restaurant where the most expensive, beautifully plated dish tastes terrible to the locals, while the simple, home-cooked meal is the favorite. The standard judges (VideoScore) were praising the wrong dish.

4. The "Magic Words" Test

The researchers also tested if the AI actually understands culture or if it just memorizes keywords.

They asked the AI: "Show me a Muslim Indian family eating." (AI gets it right).
Then they asked: "Show me a Muslim family eating." (AI gets it wrong).
The Lesson: The AI isn't truly understanding the culture; it's just looking for the word "India" or "Japan" as a trigger. If you take away the country name, the AI forgets the cultural rules. It's like a student who memorized the answer key but doesn't understand the math.

5. The Hardest Part: The "Dance"

Out of the three layers, Behavior (the actions and gestures) was the hardest for the AI to get right.

Even when the researchers gave the AI very detailed instructions, the AI still struggled to get the movement right.
The Analogy: The AI can draw a person wearing a kimono perfectly (Identity) and put them in a Japanese garden (Context), but when it tries to make them bow, it looks stiff or robotic. The "dance" of culture is still very hard for computers to learn.

Summary

The paper concludes that we cannot just trust the "pretty" scores when judging AI videos. A video can be technically perfect but culturally offensive or inaccurate. To make AI fair and useful for the whole world, we need to check the Who, the What, and the Where separately, and we need to listen to the people who actually live in those cultures, not just the machines grading the picture quality.

Technical Summary: CULTURESCORE: Evaluating Cultural Faithfulness in Video Generation Models

Problem Statement

As video generation models (e.g., Veo 3.1, LTX-2, Sora) advance in simulating the physical world, their ability to accurately represent diverse global cultures remains a critical, understudied frontier. Current evaluation metrics, such as VideoScore and UnifiedReward, focus on visual quality, temporal consistency, and factual grounding but fail to assess cultural faithfulness. These holistic metrics can be misleading; for instance, a model that replaces a culturally specific greeting (e.g., Namaste or Salam) with a generic Western handshake may receive the same high score as one that generates the correct gesture.

Existing benchmarks for cultural representation (e.g., CulturalFrames, CuRE, CUBE) have primarily focused on static text-to-image (T2I) models. However, culture is often expressed through sequential actions, gestures, and rhythms that static frames cannot capture. Consequently, there is a lack of frameworks capable of reasoning over motion sequences to evaluate whether generated videos adhere to specific cultural norms regarding identity, behavior, and context.

Methodology: The CULTURESCORE Framework

The authors propose CULTURESCORE, a compositional evaluation framework that decomposes cultural faithfulness into three granular dimensions:

Identity: Who is represented and how? This covers physical appearance, attire, and demographic markers (e.g., specific traditional clothing or professional dress norms).
Behavior: What are the people doing? This focuses on culturally normative gestures, prosody, and expressivity, requiring the evaluation of specific sequences of motion and posture (e.g., the rhythm of a bow or the hand placement in a greeting).
Context: The culturally localized background. This includes physical settings, environmental details (e.g., Islamic calligraphy, Persian rugs), and social conventions (e.g., seating arrangements).

Data Curation and Evaluation Pipeline

The framework is operationalized through a dataset grounded in CulturalFrames, adapted for video generation:

Prompts: The authors constructed an evaluation suite of 2,943 culturally validated prompts spanning 10 countries and 5 socio-cultural domains.
Prompt Augmentation: To probe model behavior, prompts were expanded into three variants:
- Base Prompt: Original culturally validated prompt.
- Extended Prompt: Augmented with explicit descriptions of Identity, Behavior, and Context derived from dictionary definitions.
- Geographical Constraint Removed: Country names are removed (e.g., "Muslim Indian family" $\to$ "Muslim family") to test if models rely on explicit geographic tokens or have internalized cultural concepts.
Video Generation: Three state-of-the-art models were evaluated: Veo 3.1 Fast (Google DeepMind), LTX-2, and Wan 2.2.
Automated Scoring (VLM-as-Judge):
- Question Generation: Using Gemini-3-Flash, the system generated 9,289 culturally grounded question-answer pairs (QA) for each video, embedding specific visual and temporal details to avoid reliance on implicit knowledge.
- Evaluation: A Vision-Language Model (Qwen3-VL-235B) analyzed the generated videos against these QA pairs to determine binary accuracy for each dimension.
- Scoring: The final CULTURESCORE is the average accuracy across the Identity, Behavior, and Context dimensions.

Human Evaluation

To validate the automated metrics, the authors conducted a study with N=45 native residents (5 per country across 9 nations) on Prolific. Participants performed:

Question Relevance: Judging if the generated QA pairs were meaningful probes.
Perceptual Accuracy: Answering the QA pairs based strictly on visual evidence.
Preference Ranking: Ranking the three models based on overall cultural faithfulness.

Key Results

1. Systematic Failure of Current Models

No current model achieves high cultural faithfulness. The best-performing model, Wan 2.2, achieved an overall CULTURESCORE of only 56.8%.

Behavior is the most challenging dimension: Across all models, Behavior scores remained below 52% (Wan 2.2: 49.8%, LTX-2: 47.0%, Veo 3.1: 51.9%).
Context performs best: Context scores were highest, with Wan 2.2 reaching 69.3%.

2. Inverse Correlation with Perceptual Metrics

A striking finding is the inverse relationship between standard video quality metrics and cultural faithfulness:

LTX-2 achieved the highest VideoScore (avg. 3.34, perceived as most "cinematic") but ranked second-to-last in CULTURESCORE (51.4%).
Wan 2.2 achieved the highest CULTURESCORE (56.8%) but received the lowest VideoScore (2.72).
This suggests that optimizing for perceptual quality (motion smoothness, visual fidelity) does not translate to cultural accuracy and may even be inversely correlated.

3. Prompt Sensitivity and Geographic Dependency

Extended Prompts: Providing explicit cultural guidance (Extended Prompts) improved scores across all dimensions, particularly for Context (+18% to +25% gains). However, Behavior remained stubbornly low, indicating that prompt enrichment alone cannot resolve temporal motion failures.
Geographic Triggers: Removing country names from prompts caused a substantial drop in accuracy (e.g., LTX-2 Context accuracy dropped from 62.1% to 28.6%). This indicates models rely heavily on explicit geographic tokens to activate cultural associations rather than having internalized cultural concepts as open-world knowledge.

4. Alignment with Human Preference

Human preference rankings aligned directionally with CULTURESCORE but were inverted relative to VideoScore:

Native evaluators ranked Veo 3.1 Fast as the most preferred (avg. rank 1.66), followed by Wan 2.2, with LTX-2 ranked last.
VideoScore incorrectly favored LTX-2.
CULTURESCORE correctly identified Wan 2.2 as the most culturally faithful among the top contenders, validating its utility as a diagnostic tool.

Contributions and Significance

The paper makes three primary contributions:

Framework Proposal: Introduction of CULTURESCORE, a compositional framework that decomposes cultural faithfulness into Identity, Behavior, and Context, enabling fine-grained diagnosis of model failures.
Benchmark Release: Construction and public release of a culturally grounded evaluation suite containing 2,943 prompts, 6,180 generated videos, and 9,289 reasoning/QA pairs across 10 countries.
Empirical Analysis: A comprehensive analysis revealing that current video generation models exhibit systematic cultural failure modes, specifically an inverse correlation between perceptual quality and cultural faithfulness.

Significance:
The authors argue that cultural faithfulness is an essential criterion for equitable access to generative AI, with implications for education, media, and policy. The study demonstrates that existing metrics can "actively mislead" model selection when cultural accuracy is the priority. By providing a diagnostic tool that exposes where and how models diverge from authentic representation (e.g., correct setting but wrong gesture), CULTURESCORE offers a foundation for auditing and improving cultural representation in generative video systems.

Limitations and Ethics

The authors acknowledge several limitations:

Asymmetric Coverage: Due to API costs, Veo 3.1 was evaluated on a stratified sample (294 videos) rather than the full suite.
Geographic Scope: Human evaluation excluded Iran due to geopolitical constraints.
VLM Bias: The automated scoring relies on VLMs (Qwen3-VL, Gemini) which may carry their own cultural biases.
Identity Definition: The operationalization of "Identity" relies on visual markers (attire, appearance), which risks reinforcing stereotypes and cannot fully capture complex concepts like lived experience or history.

The authors position CULTURESCORE as a complementary metric to human judgment, not a standalone ground truth, and emphasize that the dataset should be used to enable replication and extension rather than as a definitive account of any culture.