💬 NLP

Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

This paper applies Cultural Consensus Theory to the World Values Survey to reveal that large language models often misrepresent cultural structures by either failing to capture cohesive group consensus or over-regularizing it, thereby offering a new diagnostic framework to distinguish between genuine human diversity and algorithmic homogenization.

Original authors: Krishna Pothugunta, John P. Lalor

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Krishna Pothugunta, John P. Lalor

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, noisy library where a new kind of librarian has just arrived: a super-smart robot that reads every book ever written and can answer any question you ask. This robot is a Large Language Model (LLM). But here's the tricky part: the library isn't just one room; it's a massive building with thousands of different neighborhoods, each with its own unique customs, jokes, and ways of seeing the world. We call these neighborhoods "cultures."

For a long time, scientists have been trying to teach these robot librarians how to be polite and accurate in every neighborhood. They've mostly done this by asking the robot, "What do people in Country X think?" and then averaging the answers. It's like asking a hundred people for their favorite ice cream flavor, counting up the votes, and telling the robot, "The group likes Vanilla." But what if the group doesn't actually agree? What if half love Vanilla and half hate it, but the average says "Vanilla" anyway? That's where the real trouble starts. To fix this, researchers are borrowing a tool from the world of anthropology called "Cultural Consensus Theory." Think of this as a special magnifying glass that doesn't just count votes; it checks how much the people in the group actually agree with each other, and how much they disagree. It helps us see if a group is truly united in their thinking or just a messy collection of different opinions.

Now, imagine you have a robot librarian that is supposed to understand these neighborhoods. A team of researchers from the University of Notre Dame decided to test this robot using a giant, real-world survey called the World Values Survey, which asks people from 10 different countries about 12 different topics, like happiness, corruption, and science. They didn't just ask the robot one question; they asked it to pretend to be a person from each of those countries. Then, they used their special "agreement magnifying glass" (Cultural Consensus Theory) to compare the robot's answers against the real answers from actual humans.

What they found was a bit surprising and a little funny. The robot wasn't just "wrong" or "right"; it was behaving in three very strange ways depending on the topic.

First, in some areas, like how people feel about corruption or religion, the robot was very good at guessing the general direction of human opinion. But here's the catch: the robot was too sure of itself. It acted like everyone in the country thought exactly the same thing, with zero disagreement. The researchers call this "Consensus Inflation." It's like the robot looking at a room full of people arguing about pizza toppings and saying, "Oh, everyone clearly agrees on pepperoni!" when in reality, half the room is screaming for pineapple. The robot smoothed out all the messy, real human differences into a perfect, fake agreement.

Second, in other areas, like happiness and well-being, the robot completely failed to find any agreement at all. It was like the robot was shouting random answers that didn't match anyone in the room. The humans had a clear pattern of what they believed, but the robot couldn't find it, leaving a "Consensus Gap." It was as if the robot was trying to dance to a song it couldn't hear.

Third, and perhaps the most interesting, was the "Heterogeneity Gap." In topics like science and technology, the robot actually agreed with itself very strongly. It had a very tight, unified opinion. But that opinion didn't match the humans at all! The humans were all over the map, with some loving science and others fearing it, but the robot was confidently saying, "We all think the same thing!" and then being wrong about what that thing was.

The researchers also noticed that the robot behaved differently depending on whether it was pretending to be from a country with one main culture or a country with many mixed cultures. In the mixed-culture countries, the robot sometimes got a little better at mimicking the messiness of human disagreement, but it still struggled to get the details right.

So, what does this all mean? It suggests that while these super-smart robots are getting better at talking like humans, they are still terrible at understanding the messiness of human culture. They tend to either invent a fake, perfect agreement where none exists, or they get lost when people actually disagree. The paper doesn't say the robots are broken forever, but it does warn us that if we just look at the "average" answer, we might miss the most important part: the fact that real people are diverse, complicated, and often don't agree. The robot needs to learn to appreciate the chaos, not just the order.

Technical Summary: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theory

Problem Statement

Recent Natural Language Processing (NLP) research has begun probing Large Language Models (LLMs) for their understanding of cultural norms across different countries. However, existing approaches predominantly rely on distributional patterns and standard group-level aggregation (e.g., averaging responses). This methodology overlooks two critical dimensions:

Intra-group heterogeneity: It fails to account for dissenting opinions or variance within a specific country's population.
Multicultural environments: It often treats countries as monolithic cultural blocks, ignoring the complexity of multicultural societies.

As noted by Durmus et al. (2024), averaging responses has limitations, particularly when "people within a country have dissenting opinion." Consequently, current evaluations struggle to distinguish between models that genuinely reflect human cultural diversity and those that exhibit "algorithmic homogenization"—where models collapse diverse human views into a rigid, artificial consensus.

Methodology

To address these limitations, the authors apply Cultural Consensus Theory (CCT), a framework from cultural anthropology (Romney et al., 1986), to model multidimensional cultural nuance. CCT treats culture as a distribution of shared meanings and measures individual "cultural competence" based on agreement with a shared consensus.

Experimental Setup

Data Source: The study utilizes the World Values Survey (WVS) Wave 7, covering 10 countries and 12 cultural domains (e.g., Happiness and Well-Being, Perceptions of Corruption, Religious Values).
Country Selection: Countries were categorized into "Single-culture" (low Ethnic Fractionalization Index, e.g., Japan, Germany) and "Multi-culture" (high EFI, e.g., USA, Malaysia) groups.
Model Ensemble: Responses were generated by 10 LLMs (including GPT-4o, Llama3.1, Qwen variants, and Phi-3) using 6 different prompt variations per country to simulate human respondents.
CCT Metrics:
1. Cultural Competence ( $c_n$ ): Estimated via eigendecomposition of the agreement matrix, normalized to [0,1], representing how well a respondent (human or model) aligns with the shared cultural model.
2. Variance Explained (VE): The proportion of variance in responses explained by the first principal component (consensus).
3. Consensus Consistency (CC): The degree of matching between the LLM consensus answer and the Human consensus answer (treated as ground truth).
4. Difference in Variance ( $\Delta VE$ ): The difference between the internal consensus strength of the LLM ensemble and the human population ( $\Delta VE = VE_L - VE_H$ ).

Key Contributions

The paper introduces a fine-grained diagnostic framework that moves beyond simple accuracy to evaluate the structural rigidity of model consensus. By explicitly modeling intra-group variance, the authors provide a method to detect when models reflect true human diversity versus algorithmic homogenization.

Results

The analysis reveals three distinct regimes of model behavior across the 12 domains:

Consensus Inflation: In domains like Perceptions of Corruption (POC) and Religious Values (RV), models achieve high Consensus Consistency (CC $\ge$ 0.8) but exhibit $\Delta VE > 0$ . This indicates that while models match the direction of human consensus, they artificially amplify its strength, collapsing human diversity into a more rigid, confident consensus than exists in reality.
Consensus Gap: In domains like Happiness and Well-Being (HWB), particularly in single-culture settings, models show low CC and $\Delta VE < 0$ . Here, models fail to form a cohesive cultural alignment, exhibiting weaker or fragmented consensus compared to the human population.
Heterogeneity Gap: In domains like Perceptions of Science and Technology (POST) and Economic Values (EV), models show low competence and low CC ( $\le$ 0.5) but positive $\Delta VE$ . This suggests models converge internally on an artificial consensus that does not match the actual heterogeneity of human opinions.

Cultural Context Findings:

Domain Dependence: Model performance is highly domain-specific. For instance, Qwen2.5vl:72B showed high competence in Economic Values, while GPT-OSS:120B performed well in Perceptions of Corruption.
Single vs. Multi-Culture: The structural fit of models varies by cultural setting. For example, in the HWB domain, the gap in structural consensus fit ( $\Delta VE$ ) improved significantly (became less negative) when moving from single-culture to multi-culture settings, suggesting models handle the complexity of multicultural variance differently than single-culture variance.
Model Scale: Larger models generally achieved higher $\Delta VE$ and stronger consensus in multi-culture settings, while smaller models (e.g., Qwen 7B, Phi-3) frequently demonstrated the weakest structural fit.

Significance and Claims

The authors claim that Cultural Consensus Theory offers a theoretically grounded approach to quantify consensus strength and fragmentation, bridging the gap between population-level distribution matching and user-level personalization.

The primary significance of this work lies in its ability to provide actionable diagnostics:

It identifies specific failure modes, distinguishing between "gaps" (where models fail to align) and "inflation" (where models over-regularize human diversity).
It highlights that standard aggregation metrics can mask underlying structural misalignments, such as the "algorithmic monoculture" warned of by Zhang et al. (2025).

The paper concludes that while LLMs can match human consensus directions, they often fail to replicate the natural variance of human populations. The authors advocate for integrating CCT into prompt selection and post-training calibration to mitigate consensus inflation and heterogeneity collapse, urging the research community to move beyond broad-brush cultural evaluations toward nuanced, domain-specific assessments.

Limitations

The authors acknowledge that CCT interpretability is bounded by extreme response patterns (perfect homogeneity or total divergence), where the model cannot meaningfully estimate individual variance. Furthermore, the study excludes aggregated metrics like Hofstede's Cultural Dimensions, focusing strictly on individual-level respondent data. The Consensus Consistency metric relies on a heuristic rounding approach rather than a formal Thurstonian ordinal model.