💬 NLP

Do Large Language Models Reflect Demographic Pluralism in Safety?

This paper introduces Demo-SafetyBench, a framework designed to address the lack of demographic diversity in LLM safety alignment by reclassifying existing datasets into diverse safety domains and providing a scalable, robust method for evaluating how safety perceptions vary across different demographic groups.

Original authors: Usman Naseem, Gautam Siddharth Kashyap, Sushant Kumar Ray, Rafiq Ali, Ebad Shabbir, Abdullah Mohammad

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Usman Naseem, Gautam Siddharth Kashyap, Sushant Kumar Ray, Rafiq Ali, Ebad Shabbir, Abdullah Mohammad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Universal Rulebook" Problem: Why AI Safety Needs a Cultural Compass

Imagine you are a teacher in a massive, global school. You want to create a "Safety Rulebook" to decide what is inappropriate for students to say in class.

If you only ask teachers from one specific neighborhood to write the rules, you might end up with a book that says, "It is strictly forbidden to talk about spicy food during lunch," because that’s a local custom. But to a student from a different part of the world, that rule seems silly and unnecessary. Meanwhile, the rulebook might completely miss something truly important to another culture, like how to show respect to elders.

This paper argues that current AI models are like that one-neighborhood teacher. They are being trained on "safety rules" that mostly reflect Western or narrow cultural views. The researchers wanted to see if AI can understand that "safety" isn't a single, universal rulebook, but a collection of different perspectives depending on who you are.

The Solution: The "Demo-SafetyBench" (The Cultural Prism)

The researchers created a new testing system called Demo-SafetyBench. Instead of just asking an AI, "Is this sentence bad?", they added a "Cultural Prism" to the question.

They didn't just change the sentence; they changed the context. They took thousands of topics (like politics, drugs, or social etiquette) and attached "identity tags" to them.

The Analogy: The "Dinner Party" Test
Imagine you walk into a dinner party.

Scenario A: You are at a formal business gala in London.
Scenario B: You are at a loud, casual backyard BBQ in Texas.
Scenario C: You are at a quiet, traditional family dinner in Tokyo.

In each setting, the "safety" of a joke or a comment changes. A loud joke might be "safe" in Texas but "unsafe" (disrespectful) in Tokyo.

The researchers fed these different "settings" (Age, Race, Gender, Education) into AI models to see if the AI's "safety alarm" went off differently depending on the setting.

What Did They Find? (The Results)

They tested three different "judges" (AI models): a heavyweight champion (GPT-4o), a middleweight (LLaMA-2), and a lightweight (Gemma).

The "Big Brain" is more consistent, but not perfect: The smartest model (GPT-4o) was the most reliable. It didn't get "confused" easily, but even it still showed slight biases. It’s like a very wise judge who is mostly fair but still has some subconscious habits.
Smaller models are "Moodier": The smaller, lighter AI models were much more sensitive to demographics. Their "safety alarm" went off wildly differently depending on the person they were talking to. They are like teenagers—sometimes they follow the rules, and sometimes their judgment shifts drastically based on who is in the room.
Safety is "Socially Conditioned": Even the best AI models still have "blind spots." They don't treat every demographic group exactly the same way. This proves that AI safety isn't just about "right vs. wrong"; it's about "who is asking and from what perspective."

Why Does This Matter to You?

As AI starts helping doctors, teachers, and policymakers, we can't have a "one-size-fits-all" safety filter. If an AI is used in a conservative religious community, it needs to understand those norms. If it's used in a progressive university, it needs to understand those too.

This paper provides a scientific thermometer to measure how "culturally aware" an AI is. It helps developers build AI that doesn't just follow one set of rules, but respects the beautiful, messy, and diverse "pluralism" of the real human world.

Technical Summary: Do Large Language Models Reflect Demographic Pluralism in Safety?

1. Problem Statement

Current Large Language Model (LLM) safety alignment research often treats "safety" as a monolithic, universal construct. However, safety is inherently pluralistic; what is considered harmful or inappropriate varies significantly across different demographic, cultural, and social contexts (e.g., views on alcohol, gendered humor, or religious norms).

The authors identify two critical gaps in existing research:

Demographic Narrowness: Existing alignment datasets (like ANTHROPIC-HH or DICES) rely on annotator pools that are often demographically homogeneous or Western-centric, encoding narrow cultural priors.
Confounding Variables: Current "LLM-as-a-Judge" frameworks evaluate safety based on model responses. This mixes the model's perception of safety with its stylistic biases or refusal behaviors, making it difficult to isolate how demographic framing alone affects safety perception.

2. Methodology: Demo-SafetyBench

The authors propose Demo-SafetyBench, a two-stage framework designed to decouple value framing from model responses by modeling demographic pluralism directly at the prompt level.

Stage I: Data Construction (Prompt-Level Modeling)

Reclassification: The researchers took prompts from the DICES dataset and used Mistral-7B-Instruct-v0.3 to reclassify them into 14 safety domains (adapted from the BEAVERTAILS taxonomy), such as Hate Speech, Self-Harm, and Financial Crime. This was treated as a multi-label classification task.
Expansion: To fix class imbalance, they used Llama-3.1-8B-Instruct to synthetically generate queries for low-resource domains. Crucially, these queries were conditioned on demographic metadata (gender, race, age, education) sampled from the original dataset to maintain demographic realism.
Deduplication: They employed SimHash fingerprinting with a Hamming distance threshold ( $\tau=10$ ) to remove redundant synthetic data while preserving semantic diversity.

Stage II: Benchmarking (Pluralistic Evaluation)

LLMs-as-Raters: Three models—Gemma-7B, GPT-4o, and LLaMA-2-7B—acted as independent judges.
Zero-Shot Protocol: Each rater was given a structured input containing a prompt and its associated demographic context. The models were tasked with providing both a categorical verdict (Safe/Unsafe) and a normalized numerical confidence score ( $s \in [0, 1]$ ).
Metrics: The framework uses four key metrics to quantify pluralism:
- Demographic Sensitivity (DS): Variance in safety scores across demographic subgroups.
- Inter-Rater Correlation ( $\bar{\rho}$ ): Consistency of moral reasoning between different LLM judges.
- Fairness Gaps (DPD & EOD): Measures of demographic parity and equalized odds to detect bias.

3. Key Contributions

A Novel Framework: Introduced a method to evaluate safety pluralism at the prompt level, removing the bias introduced by model-generated responses.
A Diversified Dataset: Created a large-scale (43,050 samples) demographically grounded corpus across 14 safety domains.
Scalable Evaluation Protocol: Demonstrated that LLMs can serve as reliable, scalable, and computationally efficient judges for measuring demographic sensitivity.

4. Key Results

Model Performance & Bias:
- GPT-4o demonstrated the highest internal reliability (ICC = 0.87) and the lowest demographic sensitivity (DS = 0.119), suggesting it is the most stable and least biased rater.
- LLaMA-2-7B showed the highest demographic sensitivity (DS = 0.176) and largest fairness gaps, indicating that smaller, less-aligned models are more susceptible to demographic fluctuations.
Demographic Trends: The study found systematic variations in safety perception. For example, prompts associated with Male, White, Older (40+), and Lower-Education groups received higher safety scores (indicating stricter/more risk-averse judgments), whereas Female, Asian, Younger, and Graduate-Educated profiles were perceived as "safer."
Efficiency: The authors proved that smaller models like Gemma-7B and LLaMA-2-7B provide competitive pluralistic insights at a fraction of the computational and energy cost of GPT-4o.

5. Significance

This paper shifts the paradigm of LLM safety from "universal alignment" to "pluralistic alignment." By proving that even state-of-the-art models like GPT-4o retain residual demographic sensitivity, the research underscores that safety is not a fixed target. The work provides a vital tool for researchers to build models that are not just "safe" by a single standard, but are sensitive to the diverse moral and cultural landscapes of a global user base.