💬 NLP

Cross-Cultural Value Awareness in Large Vision-Language Models

This paper investigates how large vision-language models (LVLMs) exhibit cultural value biases by analyzing their moral, ethical, and political judgments across different cultural contexts using counterfactual image sets and Moral Foundations Theory.

Original authors: Phillip Howard, Xin Su, Kathleen C. Fraser

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Phillip Howard, Xin Su, Kathleen C. Fraser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, super-fast robots that can look at a picture and tell you a story about the person in it. These robots are called Large Vision-Language Models (LVLMs). They are like digital detectives who can see an image and instantly guess a person's personality, morals, and political views.

But here's the problem: sometimes these robots are biased. They might look at a picture of a Black man and assume he's angry, or look at a woman and assume she's not a leader. We already know they struggle with race and gender.

This paper asks a new question: What happens when we change the cultural setting around the person? Does the robot change its mind about who that person is just because they are standing in a church, a mosque, a synagogue, or a rich neighborhood versus a poor one?

The Experiment: The "Same Person, Different Stage" Trick

To test this, the researchers used a clever trick called Counterfactuals.

Imagine you have a photo of your friend, Dave.

Scenario A: You put Dave in a photo wearing a suit in a fancy office.
Scenario B: You put the exact same Dave in a photo wearing a t-shirt at a beach party.
Scenario C: You put the exact same Dave in a photo wearing a robe in a temple.

The person (Dave) hasn't changed. Only the "stage" (the background context) has changed.

The researchers took thousands of these "Same Dave, Different Stage" photos and asked five different AI robots: "What kind of values does this person hold?"

The Tools: How They Measured the Robots' Thoughts

The researchers used three special tools to analyze the robots' answers:

The Moral Compass (Moral Foundations Theory):
Think of this as a map with six different "moral territories":
- Care/Harm: Do they care about hurting others?
- Fairness/Cheating: Do they care about justice?
- Loyalty/Betrayal: Do they care about their group?
- Authority/Subversion: Do they respect leaders?
- Sanctity/Degradation: Do they care about purity or sacred things?
- Liberty/Oppression: Do they care about freedom?
The researchers checked: When Dave is in a church, does the robot say he cares about "Sanctity"? When he's in a mosque, does the robot say he cares about "Loyalty"? If the robot changes its answer based on the background, it shows cultural awareness. If it gives the exact same answer every time, it's tone-deaf.
The Sensitivity Meter (Jaccard Overlap):
This is like checking how much the robot's answers overlap. If the robot says Dave is "kind, honest, and brave" in the church, but "greedy, lazy, and angry" in the mosque, the meter goes up. This means the robot is sensitive to the context. If it says "kind, honest, and brave" for all backgrounds, the meter stays low.
The Word Dictionary (Lexical Analysis):
The researchers looked at the specific words the robots used. Do they use "warm" words (like friendly, kind) or "competent" words (like smart, capable)? They checked if the robots used different words for people in rich neighborhoods versus poor ones.

What They Found: The Robots Are Not All the Same

The study tested 5 different popular AI models. Here is what happened:

The "Chameleon" Robots (Qwen, Gemma, InternVL):
These robots were like chameleons. When they saw Dave in a church, they talked about "faith" and "purity." When they saw him in a mosque, they talked about "community" and "duty." When they saw him in a poor neighborhood, they used fewer "warm" words.
- Verdict: These robots do notice cultural differences. They are aware that context matters. However, this awareness sometimes leads them to lean on stereotypes (e.g., assuming poor people are less "competent").
The "Stuck Record" Robots (Molmo, LLaVA):
These robots were like a broken record. No matter if Dave was in a temple, a synagogue, or a slum, they gave almost the exact same list of values.
- Verdict: These robots lack cultural awareness. They seem to ignore the background clues entirely. Interestingly, one of them (LLaVA) was actually very sensitive to the background but just gave random, noisy answers, like a person guessing wildly because they don't understand the question.

The Big Takeaway

The paper reveals a hidden danger in AI.

Some robots are too aware: They see a cultural cue (like a religious building) and immediately jump to stereotypes about what people in that culture believe. They might assume a person in a mosque is "traditional" and a person in a church is "modern," even if the person is the same.
Some robots are too oblivious: They ignore the cultural context completely, treating a person in a sacred temple the same as a person in a shopping mall. This means they miss the nuance of human life.

The Conclusion:
We need to teach these AI robots to be like a wise, open-minded traveler. A good traveler sees a temple and understands it's a place of worship, but they don't immediately assume the person inside is "strict" or "old-fashioned." They know that culture is complex.

Right now, our AI robots are either too quick to judge based on the background or too blind to see it at all. This paper gives us a new way to measure that blindness and helps us build robots that understand the rich, diverse tapestry of human culture without falling into stereotypes.

1. Problem Statement

While Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal capabilities, they are increasingly criticized for reinforcing harmful societal stereotypes. Prior research has predominantly focused on demographic biases (race, gender, age) where physical appearance is the primary cue. However, there is a significant gap in understanding cultural biases related to religion, nationality, and socioeconomic status.

The core challenge is that cultural context cannot be reliably inferred from physical appearance alone; it requires interpreting environmental cues. Existing datasets often fail to isolate these cultural variables from demographic ones. Consequently, it remains unclear whether LVLMs possess genuine "cultural value awareness"—the ability to adjust their moral, ethical, and political judgments based on the depicted cultural context—or if they rely on static, potentially stereotypical priors.

2. Methodology

The authors propose a multi-dimensional evaluation framework designed to isolate the impact of cultural context on LVLM value judgments.

A. Dataset: Cultural Counterfactuals

The study utilizes the Cultural Counterfactuals dataset (Howard et al., 2026). This dataset is crucial because it contains image sets where the same person is depicted in different cultural contexts (e.g., a Christian church, a Mosque, a Hindu temple, or varying income levels).

Control: By keeping the person's appearance (race, gender, age) constant and only changing the background/context, the framework eliminates confounding variables, allowing for precise measurement of how cultural cues alone influence model output.
Scale: The study generated 2.69 million responses across five popular open-weight LVLMs (Qwen2.5-VL, Gemma-3, InternVL3, LLaVA-v1.6, and Molmo-7B) using nine unique prompts targeting moral, ethical, and political values.

B. Evaluation Framework

The authors employ three distinct analytical pillars:

Moral Foundations Theory (MFT) Analysis:
- Generated values are mapped to the six psychological foundations of MFT: Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Sanctity/Degradation, and Liberty/Oppression.
- Unique Value Filtering: To ensure the analysis captures context-specific reasoning, the authors filter for values that appear uniquely in one cultural context within a counterfactual set, excluding values common to all contexts.
- LLM-as-a-Judge: GPT-5.4 is used to categorize these unique values into the six MFT foundations.
Value Sensitivity Analysis:
- This metric quantifies how much an LVLM's generated value set changes when the cultural context changes.
- Metric: The authors use Jaccard overlap (inverse) to measure sensitivity. A higher sensitivity score indicates that the model generates significantly different values for the same person across different contexts, suggesting higher cultural awareness.
- Formula: $Sensitivity = 1 - \text{Average Jaccard Overlap}$ between value sets of different contexts within the same counterfactual set.
Lexical Analysis (Stereotype Content Model - SCM):
- Generated values are analyzed for semantic polarity using the Stereotype Content Model, which maps words to two dimensions: Warmth (Sociability/Morality) and Competence (Ability/Agency).
- This assesses whether models associate specific cultural groups with higher or lower warmth/competence scores (e.g., associating high income with higher competence).

3. Key Contributions

First Large-Scale Study on Cultural Value Bias: This work is the first to systematically investigate how LVLMs adjust moral, ethical, and political judgments based on religious, national, and socioeconomic contexts using counterfactuals.
Novel Evaluation Framework: The integration of MFT, Jaccard-based sensitivity, and SCM lexical analysis provides a robust, multi-dimensional tool for diagnosing cultural awareness, moving beyond simple accuracy metrics.
Disentanglement of Recognition vs. Reasoning: The study distinguishes between a model's ability to recognize a cultural context (classification accuracy) and its ability to reason about the associated value differences.

4. Key Results

A. MFT Findings: Variability vs. Stagnation

Context-Aware Models: Models like Qwen2.5-VL, Gemma-3, and InternVL3 exhibited significant variability in MFT category frequencies across different contexts. For example, Qwen2.5-VL associated "Care/Harm" more frequently with Christian contexts, while "Sanctity/Degradation" and "Loyalty/Betrayal" were more frequent in Hindu and Shinto contexts.
Context-Unaware Models: Molmo-7B and LLaVA-v1.6 showed almost zero variability in MFT categories across all cultural contexts. They generated nearly identical value distributions regardless of whether the person was in a Mosque or a Synagogue.
The Recognition Paradox: Interestingly, Molmo-7B and LLaVA-v1.6 had lower context classification accuracy (52-58% for religion) compared to other models (75-86%). However, even when they did classify the context correctly, they failed to adjust their value reasoning, suggesting a lack of cross-cultural reasoning capabilities rather than just poor recognition.

B. Value Sensitivity Results

High Sensitivity $\neq$ Awareness: LLaVA-v1.6 showed the highest value sensitivity (low Jaccard overlap), meaning its outputs varied wildly between contexts. However, combined with its low MFT variability and poor context classification, the authors conclude this is likely noise or over-reaction to visual features rather than genuine cultural awareness.
Balanced Awareness: InternVL3-8B and Gemma-3-12b demonstrated a "sweet spot": high context classification accuracy, high value sensitivity, and significant MFT category variability. This combination provides the strongest evidence of genuine cultural value awareness.

C. Lexical Analysis (SCM)

Socioeconomic Stereotypes: Both InternVL3-8B and Qwen2.5-VL exhibited learned stereotypes regarding socioeconomic status. As income levels increased in the images, the frequency of "Warmth" words decreased, while "Competence" words increased.
Religious Bias: Qwen2.5-VL consistently produced the lowest "Warmth" scores for the Synagogue context compared to Christian and Muslim contexts, indicating potential religious bias in its semantic associations.

5. Significance and Conclusion

This paper highlights a critical limitation in current LVLMs: while some models can recognize cultural symbols, they often fail to reason about the distinct moral and ethical value systems associated with those cultures.

Implications: The findings suggest that deploying LVLMs in cross-cultural settings without specific alignment could lead to the reinforcement of monolithic stereotypes or the erasure of cultural nuance.
Future Directions: The authors call for greater attention to instilling knowledge of value differences across cultures during model training and fine-tuning. The proposed framework serves as a diagnostic tool for developers to measure and mitigate these specific types of cultural biases before production deployment.

In summary, the study demonstrates that cultural value awareness is not a default capability of current LVLMs and requires explicit evaluation and intervention to ensure fair and culturally sensitive AI behavior.