💬 NLP

Measuring Stereotype and Deviation Biases in Large Language Models

This study investigates stereotype and deviation biases in four advanced large language models by analyzing their generated individual profiles, revealing that all examined models exhibit significant biases in associating demographic groups with specific attributes and in diverging from real-world demographic distributions.

Original authors: Daniel Wang, Eli Brignac, Minjia Mao, Xiao Fang

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Wang, Eli Brignac, Minjia Mao, Xiao Fang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have four different AI "storytellers" (Large Language Models, or LLMs). You ask them to invent a character and tell you everything about them: their political views, religion, who they love, how much money they make, and what job they have.

This paper is like a detective report that checks if these storytellers are telling the truth about the real world, or if they are just repeating old, lazy stereotypes. The researchers asked two main questions:

The "Stereotype" Test: Do these AI storytellers treat different groups of people differently? (e.g., Do they always make men "engineers" and women "teachers"?)
The "Deviation" Test: Do the characters they invent actually look like real people in the real world? (e.g., If they invent 100 people, do the political views match the actual US population, or is everyone suddenly a liberal?)

Here is what they found, broken down simply:

1. The "Political Lens" is Broken

When the AI was asked to guess a person's political party, it acted like a broken compass.

The Finding: Almost every single person the AI invented was Liberal. Whether the character was a man, a woman, Black, White, or Asian, the AI overwhelmingly said, "They are Liberal."
The Reality Check: In the real world, people are split between Liberal, Conservative, and Neutral. The AI ignored the Conservatives and Neutrals almost entirely.
The Analogy: It's like asking a chef to cook a buffet for a crowd, but the chef only serves pizza. Even if you ask for a salad, a steak, or a taco, the chef just gives you more pizza because that's what they are used to cooking.

2. The "Religion" Filter is Narrow

When the AI guessed what religion a person practiced:

The Finding: The AI mostly guessed Christian or Unaffiliated (no religion). It rarely guessed Hindu, Jewish, or Muslim, even though millions of real people follow those faiths.
The Age Twist: The AI had a specific habit with older people (Baby Boomers). It almost always guessed they were Christian. For younger generations, it guessed they were "Unaffiliated."
The Analogy: Imagine a map where the only countries drawn are the US and Canada, and every other country is just labeled "Unknown." The AI's map of the world's religions is missing huge chunks of the actual population.

3. The "Love Life" Distortion

When the AI guessed sexual orientation:

The Finding: The AI invented way too many LGBTQ+ characters. In the real world, most people identify as heterosexual. The AI flipped this, making the vast majority of its invented characters LGBTQ+.
The Gender Split: It had a specific pattern: it almost always guessed men were gay and women were bisexual.
The Analogy: It's like walking into a room where the AI says, "Everyone here is wearing a red hat," when in reality, only a few people are. The AI is amplifying a minority group to the point where it looks like the majority.

4. The "Job" and "Money" Stereotypes

When the AI guessed jobs and wealth:

The Finding: The AI relied heavily on old-school stereotypes.
- Jobs: It loved making teachers and graphic designers for almost everyone, especially older generations.
- Race & Money: It often guessed that Black people were "lower class" and Asian people were "upper class."
- Refusal: One of the AI models (Claude) got so nervous about guessing jobs for certain groups (like White men or Black men) that it simply refused to answer, saying "I can't do that."
The Analogy: The AI is like a person who has only seen a few movies. If they see a Black character, they assume they are a community organizer. If they see an Asian character, they assume they are a doctor. They are acting on "movie logic" rather than real-life statistics.

5. The "Name Game" (Implicit vs. Explicit)

The researchers did something clever. They asked the AI in two ways:

Explicit: "Write about a Black man."
Implicit: "Write about a man named Jamal." (The name implies the race without saying it).

The Result: The AI behaved almost the same way in both cases. Whether you said "Black man" directly or just used a name associated with that group, the AI still applied the same stereotypes. This suggests the AI isn't just reacting to the words you type; it has "learned" these associations deep inside its brain.

The Big Picture

The paper concludes that these AI models are not neutral mirrors of society. Instead, they are like funhouse mirrors that stretch and shrink certain groups.

They make Liberal views look like the only view.
They make LGBTQ+ identities look like the majority.
They make older people look like Christians and teachers.
They make Asian people look wealthy and Black people look poor.

The authors warn that if we use these AI models to make decisions (like hiring, lending, or analyzing news), we might be accidentally building a world that looks like the AI's distorted imagination, rather than the real world. They suggest that developers need to fix the training data so the AI stops repeating these "lazy" guesses and starts reflecting reality more accurately.

Technical Summary: Measuring Stereotype and Deviation Biases in Large Language Models

Problem Statement

Large Language Models (LLMs) are increasingly deployed in critical domains such as healthcare, law, and finance, yet they inherit and potentially amplify societal biases present in their training data. Existing literature on bias evaluation often relies on a single evaluation principle: either comparing model outputs against a human-defined "ideal" reference (normative) or against observed real-world data (descriptive). This dichotomy creates a gap; relying solely on ideal references may overlook realistic distributions, while relying solely on real-world data may reinforce existing societal inequities. Furthermore, most studies focus on explicit demographic identifiers, potentially missing biases triggered by implicit signals (e.g., names) that bypass explicit alignment mechanisms. This study addresses the need for a holistic framework that evaluates both stereotype bias (systematic, consistent associations between demographic groups and attributes) and deviation bias (disparities between LLM-generated distributions and real-world demographic statistics) across multiple social dimensions.

Methodology

The authors conducted an exploratory study involving four advanced LLMs: Claude-3.5-Sonnet, GPT-4o-mini, Command-r-plus, and Llama-3.1-70b.

Experimental Design

Prompt Strategy: The study utilized both explicit prompts (e.g., "Write a description of a Hispanic man") and implicit prompts (e.g., "Write a description of Juan," where the name implies ethnicity/gender). This design allows for the comparison of bias manifestation when demographic attributes are directly stated versus inferred.
Input Attributes: Demographics were varied across three dimensions:
- Gender: Male, Female.
- Ethnicity/Race: White, Black, Hispanic, Asian, and Neutral.
- Age: Baby Boomer, Generation X, Millennial, Generation Z, Generation Alpha.
- Note: Names were selected based on high correlation with specific demographics (e.g., top 100 US baby names for gender; ethnically specific names derived from SiSense's Ethnic Neutrality Index) to minimize noise.
Output Attributes: The models were tasked with generating 200-word profiles including attributes for:
- Political Affiliation (Conservative, Liberal, Neutral)
- Religion (Christian, Buddhist, Hindu, Jewish, Muslim, Unaffiliated)
- Sexual Orientation (Heterosexual, Homosexual, Bisexual, Other)
- Socioeconomic Status (Upper, Middle, Lower)
- Occupation
Data Generation: For implicit prompts, 500 texts were generated per gender group and 50 per ethnicity/age group. For explicit prompts, 50 texts were generated per demographic group. All generations used a temperature of 0.7.
Evaluation Metrics:
- Stereotype Bias: Measured using the mean of the maximum Jensen-Shannon (JS) divergences between demographic groups (gender, ethnicity, age) for a given output attribute. Higher values indicate greater disparity in how the model treats different groups.
- Deviation Bias: Calculated as the proportion of output attributes where the LLM-generated distribution significantly deviates from real-world US population statistics (using binomial tests with $p < 0.05$ ). Real-world data was sourced from Pew Research Center, Gallup, and other reputable demographic studies.
- Polarity: Sentiment analysis was performed to measure the valence (positive/negative) of generated texts.

Key Contributions

Dual-Framework Evaluation: The paper introduces a comprehensive evaluation framework that simultaneously assesses stereotype bias (internal consistency across groups) and deviation bias (alignment with reality), addressing the limitations of prior studies that use only one reference type.
Implicit vs. Explicit Bias Analysis: By systematically comparing implicit (name-based) and explicit (attribute-based) prompts, the study reveals that bias patterns remain largely consistent regardless of prompt type, though explicit prompts sometimes trigger higher refusal rates or more extreme deviations.
Multi-Dimensional Scope: Unlike many prior works focusing on single attributes (e.g., only gender or only toxicity), this study evaluates bias across five distinct social dimensions: politics, religion, sexual orientation, socioeconomic status, and occupation.
Quantitative Bias Metrics: The authors provide standardized metrics (Stereotype Bias Score and Deviation Bias Score) for four major LLMs, offering a baseline for future comparative analysis.

Key Results

Political Affiliation

Liberal Overrepresentation: All four models overwhelmingly classified individuals as "Liberal" (often >90% or 100%), significantly deviating from real-world distributions.
Underrepresentation of Neutrality: "Neutral" political affiliations were drastically underestimated (often <10%).
Age Exception: Models (except GPT-4o-mini) tended to associate Baby Boomers with "Conservative" views, reflecting a specific stereotype.
Bias Consistency: Bias patterns were consistent across implicit and explicit prompts.

Religion

Christian/Unaffiliated Dominance: Models predominantly assigned "Christian" or "Unaffiliated" labels.
Minority Underrepresentation: Non-Christian religions (Hindu, Jewish, Muslim) were severely underrepresented or assigned near-zero probabilities, despite real-world prevalence.
Demographic Correlations: Models exhibited strong stereotypes, such as associating White individuals with Judaism and Black individuals with Islam, which often diverged from actual demographic distributions.

Sexual Orientation

LGBTQ Overrepresentation: All models significantly overrepresented sexual minorities (Homosexual, Bisexual) compared to real-world statistics (where ~7.6% identify as LGBTQ).
Gender Stereotypes: Models frequently associated females with "Bisexual" and males with "Homosexual."
Deviation: Deviation bias scores were extremely high (close to 1.0) for sexual orientation across all models, indicating a systematic failure to align with real-world distributions.

Socioeconomic Status (SES)

Class Stereotypes: Models tended to output "Middle-class" and "Lower-class" statuses.
Racial Disparities: Black individuals were disproportionately associated with "Lower-class" status, while Asian individuals were often associated with "Upper-class" status, reflecting specific societal stereotypes.
Refusal Rates: Claude-3.5-Sonnet exhibited high refusal rates for prompts involving White and Black males, and Asian females, complicating direct comparison for these groups.

Occupation

Stereotypical Assignments: "Teacher" was the most common occupation assigned to Baby Boomers across all models.
Gender and Ethnicity: Models assigned "Engineer" to males (especially in Claude-3.5-Sonnet) and "Social Worker" or "Teacher" to females. Specific ethnic stereotypes emerged, such as "Community Organizer" for Black and Hispanic individuals.

Polarity (Sentiment)

Explicit vs. Implicit: Texts generated from explicit prompts showed larger variations in sentiment scores across demographic groups compared to implicit prompts.
Marginalized Groups: Explicit prompts often resulted in higher polarity scores (more positive sentiment) for historically marginalized groups (Black, Hispanic, Female), while White, Asian, and Male groups received lower scores, suggesting a potential "over-correction" or alignment bias in explicit scenarios.

Significance and Claims

The authors claim that their findings highlight the significant risks of LLMs in real-world applications where user attributes are inferred or explicitly stated.

For Users: The study underscores the necessity of critical engagement with model outputs, particularly in sociopolitical analysis, as models may present biased distributions as objective facts.
For Developers: The consistent overrepresentation of certain attributes (e.g., Liberal politics, LGBTQ identities, Christian religion) suggests that current RLHF (Reinforcement Learning from Human Feedback) processes may inadvertently encode specific societal biases or "alignment" preferences that diverge from reality.
For Organizations: The results indicate that deploying LLMs without understanding these biases can lead to skewed decision-making or the reinforcement of harmful stereotypes.

The paper concludes that while LLMs are powerful tools, their current behavior in inferring demographic attributes reveals deep-seated stereotype and deviation biases. The authors advocate for careful prompt design, model selection, and further research into balancing inclusivity with accurate representation of real-world distributions. They modestly note that these biases may evolve with model updates and that future work should explore diverse ethnic groups and alternative reference distributions beyond US-centric data.