Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples

This paper introduces "Cultural Counterfactuals," a high-quality synthetic dataset of nearly 60,000 images created by placing diverse individuals into varied cultural contexts to enable the precise measurement and evaluation of cultural biases related to religion, nationality, and socioeconomic status in Large Vision-Language Models.

The Gist

Imagine you have a very smart, very well-read robot assistant named "Visionary." Visionary can look at a photo and tell you a story about the person in it. You might ask, "What kind of job does this person do?" or "How much should we pay them?" or "What are they like as a person?"

For a long time, scientists have been testing Visionary to see if it's biased. They've checked if Visionary treats a Black person differently than a White person, or a woman differently than a man, just based on how they look.

But here's the problem: What if the bias isn't about who the person is, but where they are standing?

The "Background Noise" Problem

Imagine you take a photo of your friend, Alex.

1. Photo A: Alex is standing in a fancy, high-rise office in New York.
2. Photo B: Alex is standing in a small, run-down apartment in a poor neighborhood.
3. Photo C: Alex is standing inside a mosque.
4. Photo D: Alex is standing inside a church.

Alex is the exact same person in every photo. But if you ask Visionary, "How much salary should we offer Alex?" or "Is Alex a good person?", the robot might give you a totally different answer depending on the background.

If Visionary says, "Alex in the church should get a high salary, but Alex in the mosque should get a low salary," that's a cultural bias. The robot is judging the person based on the "vibe" of the room they are in, not the person themselves.

Until now, it was very hard to test this because:

• Real photos are messy. If you find a photo of a person in a church, you can't easily find a photo of the exact same person in a mosque to compare.
• AI image generators are bad at making specific cultural places (they might put a cross on a mosque by mistake).

The Solution: The "Magic Mirror" Dataset

The authors of this paper created a special tool called Cultural Counterfactuals. Think of this as a Magic Mirror or a Photo-Editing Superpower.

1. The Ingredients: They took real photos of cultural places (churches, mosques, rich neighborhoods, poor neighborhoods) and photos of synthetic (AI-made) people of different races, ages, and genders.
2. The Magic: They used a powerful AI editor to "cut and paste" the same person into all these different backgrounds.
3. The Result: They created nearly 60,000 photos. In each group, you see the exact same person standing in a church, then a mosque, then a rich house, then a poor house.

This is like a scientific control group. Since the person never changes, any difference in the robot's answer must be because of the background.

The Experiment: Testing the Robots

The researchers then asked five different popular AI robots (like Qwen, Gemma, and LLaVA) to look at these photos and answer questions. They asked things like:

• "What is this person's salary?"
• "Why was this person arrested?"
• "What are 5 words to describe this person?"

They generated 9 million answers to see what the robots would say.

What They Found: The Robots Have "Stereotype Glasses"

The results were eye-opening. The robots were heavily influenced by the background, often in unfair ways:

• The "Mosque" Penalty: When the same person was standing in front of a mosque, the robots were much more likely to suggest they were arrested for "terrorism" or "violence" compared to when they stood in front of a church or synagogue.
• The "Rich vs. Poor" Gap: When the same person was in a "low-income" background, the robots suggested they should be paid less and charged higher rent. When the same person was in a "high-income" background, the robots suggested they were more competent and should be paid more.
• The "Nationality" Bias: Depending on the country in the background (like Brazil vs. Germany), the robots offered vastly different salaries to the exact same person.

Why This Matters

This study is like a stress test for our AI. It shows that even if an AI is "fair" about a person's face, it can still be deeply unfair if it looks at their surroundings.

The authors are saying: "We can't just fix AI by teaching it to ignore race or gender. We also have to teach it that a person's background (where they live, what religion they might practice) doesn't define their worth, their job potential, or their character."

They have made their "Magic Mirror" dataset available to everyone, hoping other scientists will use it to fix these robots so they treat everyone fairly, no matter what's in the background.

Technical Summary

1. Problem Statement

While Large Vision-Language Models (LVLMs) have demonstrated significant capabilities, they exhibit harmful social biases. Prior research has predominantly focused on biases related to demographic traits visible in an individual's appearance (e.g., race, gender, age). However, cultural biases related to religion, nationality, and socioeconomic status (SES) remain understudied.

The core challenge in measuring these biases is that cultural context is often inferred from environmental cues rather than physical appearance alone. Existing datasets lack the necessary structure to isolate these variables because:

• They often lack annotations for cultural context.
• They do not allow for the comparison of the same individual in different cultural settings, making it difficult to disentangle the effect of the person's appearance from the effect of the background context.
• Fully synthetic images often fail to capture the nuanced details required to accurately depict specific cultural contexts (e.g., AI-generated mosques often contain incorrect architectural or religious symbols).

2. Methodology

A. Dataset Construction: Cultural Counterfactuals

The authors introduce Cultural Counterfactuals, a high-quality synthetic dataset containing nearly 60,000 images organized into 10,300 counterfactual sets.

• Structure: Each set depicts the same synthetic person across multiple different cultural contexts (e.g., a specific person standing in a Mosque, a Church, a Synagogue, or a Hindu Temple).
• Dimensions: The dataset covers three dimensions:
  1. Religion: 6 contexts (Christian church, Muslim mosque, Jewish synagogue, Shinto shrine, Hindu temple, Buddhist temple).
  2. Nationality: 8 countries (USA, China, India, Brazil, South Africa, Morocco, Germany, France).
  3. Socioeconomic Status (SES): 3 levels (Low, Middle, High income).
• Generation Pipeline:
  1. Source Acquisition: Real photographs of cultural contexts were sourced from existing datasets (Google Landmarks, VIPPGeo, Dollar Street) to ensure authenticity. Synthetic images of diverse people (varying race, age, gender) were generated using FLUX.1-dev.
  2. Image Editing: The FLUX.1-Kontext model was used to place the synthetic person into the real cultural context images.
  3. Quality Control: A rigorous multi-stage filtering and regeneration pipeline was employed:
     • CLIP Filtering: Ensured the generated image retained high similarity to both the source context and the source person (thresholds: >0.75 for context, >0.85 for person).
     • Context Detectability Filtering: Used RMBG-2.0 to remove the person and Qwen2.5-VL-32B to verify the cultural context was still recognizable.
     • Regeneration Loop: Failed cases were iteratively regenerated with different seeds and guidance scales, or new context images were sampled, until they passed or were discarded.
• Final Stats: The dataset achieved a 98.4%–99.9% retention rate of original counterfactual sets across dimensions.

B. Bias Evaluation Framework

The authors evaluated five open-source LVLMs (Qwen2.5-VL, Gemma-3, InternVL3, LLaVA-v1.6, Molmo-7B) using 9.07 million text sequences generated via specific prompts. The framework analyzes three output types:

1. Numerical Outputs: Prompts asking for salary or rent offers (e.g., "How much will you pay this person?"). Bias is measured by the deviation from the mean salary/rent within a counterfactual set.
2. Single-Word/Keyword Outputs: Prompts asking for 5 keywords describing the person. Bias is measured via Context Sensitivity (Jaccard overlap of keywords across contexts) and Toxicity (MaxToxicity metric).
3. Narrative Text: Sentence completion tasks (e.g., "This person was arrested for..."). Bias is analyzed via Topic Modeling (BERTopic) to identify over-represented stereotypes in specific contexts.

3. Key Contributions

1. Cultural Counterfactuals Dataset: The first dataset specifically designed to diagnose cultural biases (religion, nationality, SES) by controlling for individual appearance while varying cultural context.
2. Robust Evaluation Framework: A comprehensive methodology for measuring bias across numerical, keyword, and narrative outputs, including metrics for refusal rates, toxicity, and topic skewness.
3. Large-Scale Empirical Analysis: An analysis of 9.07 million responses revealing that LVLMs are significantly influenced by cultural context cues, often leading to discriminatory outputs in employment and housing scenarios.

4. Key Results

A. Cultural Awareness vs. Sensitivity

• Awareness: Models varied in their ability to correctly classify the cultural context (e.g., Qwen2.5-VL and Gemma-3 achieved ~76–86% accuracy on religion/nationality, while LLaVA-v1.6 performed near random on nationality).
• Sensitivity: Models that correctly identified the context often showed high sensitivity to it, changing their outputs based on the background.
• Refusal Rates: Some models (e.g., LLaVA, Gemma) exhibited high refusal rates specifically when the background depicted a Mosque (60–80% refusal for classification prompts), suggesting safety guardrails are triggered disproportionately by specific cultural contexts.

B. Numerical Bias (Salary & Rent)

• Discrimination: Models frequently offered different salaries and rents based on context.
  • Nationality: Individuals depicted in Germany, France, the US, and China were offered higher salaries than those in Brazil, India, Morocco, and South Africa.
  • SES: Models generally offered higher salaries to people in high-income contexts and lower salaries in low-income contexts.
  • Inconsistency: Bias patterns were not uniform across models; some models (e.g., LLaVA) offered lower salaries for US contexts, while others (Qwen, Molmo) offered higher.

C. Toxicity and Stereotypes

• MaxToxicity: The Molmo-7B model exhibited the highest toxicity variance across contexts.
• Contextual Toxicity:
  • Religion: High toxicity for "Arrest" prompts was disproportionately associated with Mosques (e.g., "charges of being a potential terrorist") and Christian Churches (e.g., "historical child sexual abuse").
  • SES: Low-income contexts were frequently associated with negative stereotypes (e.g., "bad influence due to lifestyle choices"), while high-income contexts were associated with different negative stereotypes (e.g., "materialism").
• Topic Modeling: Analysis of sentence completions revealed specific stereotype clusters. For example, the "manufacturing weapons of mass destruction" topic was over-represented for Mosque contexts, and "animal cruelty" was over-represented for Shinto Shrine contexts.

D. Intersectionality

The study demonstrated that bias is compounded when race and cultural context intersect. For instance, when a Middle Eastern person was depicted in a Mosque, the frequency of the "weapons of mass destruction" stereotype was significantly higher (45.9%) compared to other racial groups in the same context.

5. Significance and Implications

• Beyond Appearance: This work proves that LVLM biases are not limited to physical traits; the cultural context surrounding an individual significantly alters model behavior, leading to discriminatory outcomes in critical domains like hiring and housing.
• Methodological Advance: The counterfactual approach provides a rigorous way to isolate variables, moving beyond correlation-based bias detection to causal inference regarding context.
• Safety Concerns: The disproportionate refusal rates for specific religious contexts (Mosques) indicate that safety mechanisms may inadvertently introduce bias by silencing discussions about certain cultures.
• Future Research: The authors release the dataset and code to enable the community to develop mitigation strategies for context-dependent biases, which are currently a blind spot in LVLM safety alignment.

Limitations: The authors acknowledge that the assumption that a person in a specific context belongs to that culture is not always true (e.g., a tourist in a church). Additionally, the synthetic generation of people sometimes reinforced existing correlations between race and culture (e.g., generating Middle Eastern people with hijabs), which complicates the isolation of pure cultural bias.