Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed as automated annotators for text classification in academic research, content moderation, and hiring. While prior research has documented racial bias in generative outputs and single-task classifiers, there is a critical gap in understanding how implicit identity cues (such as names or dialect) influence zero-shot annotation outcomes across a wide range of dimensions. The paper investigates whether LLMs systematically reproduce racial stereotypes when labeling text, potentially embedding socially patterned biases directly into the datasets and measurements that underpin research and decision-making.

2. Methodology

The study conducted a large-scale audit of 19 LLMs from nine organizations (including OpenAI, Anthropic, Google, Meta, and Chinese providers) across the US, Europe, and China. The research utilized two complementary experimental paradigms involving over 4 million annotation judgments.

Experiment 1: The Names Paradigm

• Design: 211 biographical text vignettes were created. In each iteration, models annotated pairs of texts that were identical in content but differed only in the embedded first name.
• Stimuli: Names were selected from a US voter registration dataset to maximize distinctiveness for four minority groups (Black, Asian, Arab/Middle Eastern, Hispanic) compared to White names. Distinctiveness was verified using text embeddings and cosine similarity against group-specific anchor vectors.
• Tasks: 39 binary classification tasks spanning employability, cognition, personality, character traits, and negative stereotypes (e.g., "Is this person aggressive?", "Is this person hireable?").
• Scale: 16 models completed 100 iterations; 3 models completed 50. Total responses: ~4.01 million.

Experiment 2: The Dialect Paradigm

• Design: Matched text pairs were presented in Standard American English (SAE) and African American Vernacular English (AAVE). The propositional content was identical; only the dialect features (e.g., habitual be, copula deletion, negative concord) differed.
• Tasks: 16 annotation tasks, including the 14 from Experiment 1 plus "toxicity" and "anger" for message-style texts.
• Scale: All 19 models completed 50 iterations per task. Total responses: ~144,402.

Statistical Analysis

• Metrics: The primary outcome was the mean annotation gap (Minority/AAVE "yes" rate minus White/SAE "yes" rate).
• Inference: One-sample t-tests against zero were used, with Benjamini–Hochberg False Discovery Rate (FDR) correction applied across the full matrix of tests (19 models × 39/16 tasks × 4 groups).
• Robustness: Directional consistency (fraction of models showing the same sign) was analyzed alongside statistical significance.

3. Key Contributions

• Scale and Scope: This is the largest audit of its kind, covering 19 diverse models and 55 total annotation tasks, moving beyond single-model or single-task studies.
• Dual-Paradigm Approach: By comparing names (explicit identity markers) vs. dialect (implicit identity markers), the study isolates how different types of identity cues trigger bias.
• Identification of "Overcorrection": The study reveals a counter-intuitive finding where models systematically favor minority-named applicants in hiring tasks, suggesting that instruction tuning may have overcorrected for explicit name-based discrimination while leaving deeper stereotype structures intact.
• Universal Dialect Bias: It demonstrates that dialect-based bias (AAVE vs. SAE) is a shared property across all tested models and organizations, indicating a systemic issue in the current generation of LLMs rather than a specific training artifact.

4. Key Results

A. Names-Based Stereotypes (Experiment 1)

Bias patterns were group-specific and multidimensional, mirroring known human stereotypes:

• Black Names: Rated as more aggressive (18/19 models) and gossipy (18/19). However, they were rated as more hireable than White names (a reversal of human audit studies).
• Asian Names (The "Bamboo Ceiling"): Exhibited a distinct profile of cognitive elevation (more intelligent, creative) but social/agency devaluation (less confident, less sociable, less leader-like). This mirrors the "bamboo ceiling" stereotype.
• Arab Names: Showed cognitive elevation (intelligent, ambitious) alongside interpersonal devaluation (less warm, less good communicators) and negative character attributions (more emotional, lazier).
• Hispanic Names: Associated with cognitive elevation but economic disadvantage (more likely to be poor) and leadership penalties.
• Universal Penalty: Across all four minority groups and all 19 models, self-discipline was the most consistently penalized attribute.

B. Dialect-Based Bias (Experiment 2)

When identity was conveyed via dialect (AAVE) rather than names, the bias shifted to near-universal disadvantage across all models:

• Professionalism & Education: All 19 models rated AAVE texts as significantly less professional (mean gap −0.774) and less indicative of an educated speaker (−0.688) compared to SAE.
• Negative Traits: AAVE texts were rated as more toxic (18/19 models) and more angry (19/19).
• Hireability: Unlike the name-based experiment, AAVE speakers were rated as less hireable across all 19 models (mean gap −0.305).
• Conclusion: The dialect experiment suggests that while models may have been tuned to avoid explicit name-based hiring discrimination, they retain deep-seated biases against non-prestige dialects.

5. Significance and Implications

• Reproduction of Social Hierarchies: The study demonstrates that using LLMs for automated annotation does not merely reflect data but actively operationalizes and stabilizes social hierarchies. The biases are not random noise but follow recognizable, historically sedimented stereotype structures.
• The "Overcorrection" Trap: The finding that models favor minority names in hiring tasks while penalizing them on character traits (like self-discipline) or dialect suggests that current safety tuning (instruction tuning) may be superficial. It corrects the most visible, auditable bias (name-based hiring) while leaving implicit biases (dialect, character traits) untouched or even exacerbated.
• Research Validity: Researchers using LLMs for text annotation must treat model judgments with extreme caution. Standard validation (comparing to human gold standards on small samples) is unlikely to reveal these systematic distortions because the biases are implicit and span multiple dimensions simultaneously.
• Systemic Nature: The consistency of results across models from nine different organizations (US, Europe, China) indicates that dialect-based devaluation of AAVE is a fundamental property of the current LLM generation, likely stemming from the scale and nature of the training data rather than specific model architectures.

Conclusion: The paper concludes that LLMs, when used as automated annotators, act as "bias amplifiers" that embed racial stereotypes into the very metrics used for research and governance. The study calls for a re-evaluation of how automated text analysis is conducted and warns against relying on LLMs for sensitive classification tasks without rigorous, multi-dimensional bias auditing.