💬 NLP

ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs

This paper introduces ValueGround, a novel benchmark that evaluates the ability of multimodal large language models to align culture-conditioned value judgments with visual representations, revealing a significant performance drop when moving from text-only to visualized response options.

Original authors: Zhipin Wang, Christoph Leiter, Christian Frey, Mohamed Hesham Ibrahim Abdalla, Josif Grabocka, Steffen Eger

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhipin Wang, Christoph Leiter, Christian Frey, Mohamed Hesham Ibrahim Abdalla, Josif Grabocka, Steffen Eger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human culture. You ask it, "In Germany, do people think it's important for children to be unselfish?" If you give the robot the answer choices as words ("Yes" or "No"), it might get it right. But what if you show it two pictures instead? One picture shows a child sharing a toy; the other shows a child keeping all the toys. Can the robot still figure out that, culturally, Germany leans toward the "sharing" picture?

This is exactly what the paper ValueGround is about. It's a new test designed to see if AI models can understand cultural values when the answers are pictures instead of words.

Here is the story of the paper, broken down into simple concepts:

1. The Problem: The "Text Trap"

For a long time, researchers tested AI on culture using only text. They asked, "What would a person in Japan think about X?" and the AI answered with words.

The Flaw: Humans don't just learn culture from books; we learn it by seeing things. We see how people greet each other, how they eat, and how they behave in crowds.
The Surprise: The researchers found that when they switched the answers from words to pictures, the AI got confused. It would say one thing when reading the text, but flip its answer when looking at the images. It's like a student who can recite the rules of soccer perfectly but freezes when they actually step onto the field.

2. The Solution: A "Spot the Difference" Game

To fix this, the team built a benchmark called ValueGround. Think of it as a high-stakes game of "Spot the Difference," but the differences are about deep cultural values.

The Setup: They took real survey questions from the World Values Survey (like "Is it important to be honest?").
The Twist: Instead of giving the AI text options, they generated two very similar images.
- Image A might show a person helping a neighbor.
- Image B might show the same person walking past the neighbor.
- Crucial Rule: The images had to be almost identical (same background, same people, same clothes) so the AI couldn't cheat by looking for easy clues like flags or text. The only difference had to be the action representing the value.
The Task: The AI is shown a country (e.g., "Brazil"), the question, and the two pictures. It has to pick the picture that best matches what people in Brazil usually value.

3. How They Made the Pictures (The "Robot Chef")

Making these pictures was hard. If you ask an AI to draw "honesty," it might draw a person holding a sign that says "I am honest." That's cheating!

The Team's Secret Sauce: They used a team of "AI Agents" (like a robot kitchen staff) to build the images:
1. The Planner: Decides the scene (e.g., "Two people at a market").
2. The Editor: Takes a base photo and makes tiny, precise changes (e.g., "In Picture A, the person hands over money; in Picture B, they keep it").
3. The Critic: Checks the work. "Wait, did you accidentally put a 'Yes' sign in the background? Delete it. Make sure the clothes are the same."
  This ensured the images were fair and focused only on the cultural value.

4. The Results: The AI Got Stuck

When they tested six of the smartest AI models available today, the results were revealing:

Text Mode: The AI was pretty good (about 73% accuracy). It knew the cultural stereotypes.
Image Mode: The AI dropped to about 66% accuracy.
The "Flip": The most interesting finding was that for many questions, the AI would get the answer right with text, but get it wrong with images. It's as if the AI has two different brains: one that knows cultural facts, and another that gets confused when it has to connect those facts to a visual scene.

5. Why This Matters

This paper is a wake-up call. It shows that while AI is getting better at "seeing," it still struggles to connect what it sees with what it knows about the world.

The Metaphor: Imagine an AI that knows the definition of "politeness" perfectly but, when it sees a picture of someone bowing, it thinks it's a sign of weakness.
The Future: We need AI that doesn't just memorize cultural facts but can "ground" them in reality—understanding that culture is lived, seen, and felt, not just read.

In a nutshell: The paper built a new test to see if AI can understand culture through pictures. It found that while AI is good at reading about culture, it often gets lost when trying to see it, proving that there is still a big gap between knowing a fact and understanding a visual reality.

1. Problem Statement

Current evaluations of cultural values in Large Language Models (LLMs) are almost exclusively text-only. Models are typically tested on whether they can predict cultural tendencies based on survey questions and verbal response options (e.g., "Yes" vs. "No"). However, cultural values are often expressed through visual scenes and social practices, not just abstract language.

The core problem addressed by this paper is cross-modal instability: It is unclear whether Multimodal Large Language Models (MLLMs) can maintain consistent, culture-conditioned judgments when response options are presented as images rather than text. The authors hypothesize that models may fail to ground value judgments in visual evidence, leading to prediction reversals where a model answers correctly in text but incorrectly when the same options are visualized.

2. Methodology

A. The ValueGround Benchmark

The authors introduce ValueGround, a benchmark derived from the World Values Survey (WVS).

Task Formulation: Given a country ( $c$ ), a survey question ( $q$ ), and a pair of images ( $I_A, I_B$ ) representing opposing response options, the model must select the image that best aligns with the cultural value tendency of that country.
Constraint: The model is not provided with the original text of the response options during the test; it must infer the meaning solely from the images.
Data Construction: The benchmark covers 13 countries and utilizes 224 carefully constructed image pairs derived from WVS questions.

B. Agentic Paired-Image Construction Framework

To ensure the images are valid test cases, the authors developed an agentic pipeline to generate "minimally contrastive" image pairs. This framework uses four components to satisfy three design principles:

Semantic Alignment: Each image must clearly reflect its specific option meaning.
Minimal Contrast: The two images must share the same base scene, characters, and framing, differing only in the specific attribute that defines the option contrast.
Shortcut Suppression: Images must not contain explicit text, flags, checkmarks, or UI elements that trivially reveal the answer.

The Pipeline Agents:

Planner Agent: Analyzes the question and options to create a structured plan for a shared base scene and identifies which attributes need to be edited to create the contrast.
Base Image Generator: Creates a neutral base image based on the shared scene plan.
Editor Agent: Performs constrained edits on the base image to generate the two distinct options ( $I_A$ and $I_B$ ), ensuring minimal variation outside the target semantic axis.
Critic Agent: Evaluates the generated pair against the rubric (alignment, contrast, shortcuts). If the pair fails, it triggers a loop for regeneration, re-editing, or replanning.

C. Evaluation Settings

The authors evaluate MLLMs across three settings to isolate the source of errors:

Main Task (Visual Grounding): Input = (Country, Question, Image Pair). Output = Choice of Image A or B.
Text-Only Setting: Input = (Country, Question, Text Options). Output = Choice of Text Option A or B. (Tests cultural priors).
Option-Image Alignment Setting: Input = (Question, Text Options, Image Pair) without country context. Output = "Aligned" or "Swapped". (Tests if the model can visually distinguish the options).

3. Key Contributions

New Benchmark: ValueGround is the first benchmark to evaluate culture-conditioned value grounding where response options are visualized, moving beyond text-only cultural alignment.
Agentic Construction Pipeline: A novel, human-validated framework for generating high-quality, minimally contrastive image pairs that control for irrelevant visual variables.
Empirical Findings on Cross-Modal Reversals: The paper provides evidence that MLLMs suffer from significant performance drops and prediction reversals when moving from text to visual modalities, even when the visual contrast is clear.

4. Key Results

A. Performance Drop

Across six state-of-the-art MLLMs (including GPT-5 Mini, Gemini 3 Flash, Claude Haiku 4.5, etc.) and 13 countries:

Text-Only Accuracy: Average 72.8%.
Visual Alignment Accuracy: Average 92.8% (Models can easily distinguish the visual difference between the two images).
Main Task (Visual Grounding) Accuracy: Average 65.8%.
Conclusion: Despite high visual alignment scores, accuracy drops significantly when the model must link the visual contrast to a specific country's cultural context.

B. Prediction Reversals

Cross-Modal Reversals: A significant portion of instances (ranging from 12.1% to 36.9% depending on the model) show a reversal where the model answers correctly in the text-only setting but incorrectly in the visual setting (or vice versa).
Harmful vs. Beneficial: Reversals are predominantly "harmful" (correct text $\to$ incorrect image), indicating that visual grounding introduces noise or bias that disrupts the model's cultural priors.
Model Robustness: Stronger models (e.g., Gemini 3 Flash, GPT-5 Mini) are more robust but still exhibit reversals. Weaker models (e.g., Claude Haiku 4.5) show the largest performance gaps.

C. Difficulty Analysis

Country Variance: Performance varies by country. Great Britain and the US were the easiest; Russia and Mexico were the hardest.
Bottleneck: The primary difficulty is not recognizing the visual contrast (which is easy) nor lacking cultural knowledge (text-only performance is decent), but integrating country context, question semantics, and visual evidence into a coherent decision.

5. Significance and Implications

Limitations of Current MLLMs: The study reveals that MLLMs do not possess robust, modality-invariant cultural reasoning. Their value judgments are unstable when the representation of options shifts from abstract text to concrete visual scenes.
Beyond Textual Bias: It highlights that cultural alignment fine-tuning on text may not translate to real-world scenarios where values are encountered visually (e.g., in media, social interactions, or user interfaces).
Diagnostic Tool: ValueGround serves as a controlled testbed to disentangle textual priors, shallow visual cues, and genuine multimodal grounding.
Future Research: The paper calls for research into improving the cross-modal transfer of cultural reasoning and developing models that can ground abstract social values in visual evidence without relying on superficial shortcuts.

In summary, ValueGround demonstrates that while MLLMs are improving at cultural text analysis, they struggle significantly to apply that cultural understanding when the "answers" are visualized, revealing a critical gap in their multimodal reasoning capabilities.