Digital Skin, Digital Bias: Uncovering Tone-Based Biases in LLMs and Emoji Embeddings

1. Problem Statement

As Large Language Models (LLMs) increasingly mediate online communication, they act as the foundational infrastructure for interpreting digital symbols, including emojis. While Unicode introduced skin-tone modifiers (based on the Fitzpatrick Scale) in 2015 to enhance inclusivity, it remains unclear whether AI models represent these variants equitably.

The paper addresses two critical gaps:

1. Representation Gaps: Legacy static embedding models (trained before 2015 or on limited data) may lack vocabulary support for skin-toned emojis entirely.
2. Latent Bias: Even in models that support skin tones, there is no guarantee that the learned representations are semantically consistent or free from societal biases (e.g., associating darker skin tones with negative sentiment or specific stereotypes).

The core research question is: Do skin-tone variants receive consistent, equitable treatment in representation and downstream associations across static embedding models and modern LLMs?

2. Methodology

The authors conducted a large-scale comparative study involving nine static embedding models and four modern LLMs (Llama, Gemma, Qwen, Mistral). The methodology was structured into three stages:

A. Model Coverage and Tokenization Analysis

• Static Models: Benchmarked models like emoji2vec, emoji-sw2v, FastText, GloVe, and BERT to determine vocabulary coverage of skin-toned emojis.
• LLM Tokenization: Analyzed how subword tokenizers (BPE, SentencePiece) decompose skin-tone emojis. The study measured token counts for the five skin-tone modifiers to identify "computational bias" (i.e., whether certain tones require more tokens, increasing cost and latency).

B. Semantic Consistency Evaluation

To measure if skin tone modifiers alter the core meaning of an emoji, the authors employed:

• Emoji-Text Alignment: Calculated Cosine Distance (for aggregated vectors) and Word Mover's Distance (WMD) (for token sequences) between an emoji and its textual description (e.g., "OK hand").
• Variant Clustering: Used t-SNE visualization to observe how different skin-tone variants of hand gestures cluster in semantic space. Do they cluster by meaning or by skin tone?
• Pairwise Similarity: Generated heatmaps of pairwise cosine distances between different skin tones of the same base emoji to detect systematic semantic drift.

C. Bias Quantification

The study adapted three established text-bias metrics for the emoji domain:

1. Relative Norm Distance (RND): Measures if neutral words are closer in vector space to light-toned or dark-toned emoji groups.
2. Word Embedding Association Test (WEAT): Quantifies implicit associations between skin-tone groups and attribute sets (e.g., "Good" vs. "Bad" emojis, or societal concepts like gender/career).
3. Relative Negative Sentiment Bias (RNSB): Uses a logistic regression classifier trained on sentiment labels to determine if negative sentiment is disproportionately distributed across specific skin tones.

3. Key Contributions

1. First Cross-Family Benchmark: Provided the first comprehensive comparison of skin-tone support across nine static models and four LLMs, revealing that legacy models suffer from severe coverage gaps.
2. Tokenization-Level Asymmetry: Discovered a novel form of bias where specific models (notably Mistral) require significantly more tokens to represent dark skin tones compared to light tones, leading to higher computational costs and latency for specific demographics.
3. Semantic Drift Evidence: Demonstrated that even in models with full support, skin-tone modifiers induce consistent semantic shifts, causing emojis to drift from their textual descriptions or cluster by skin tone rather than meaning.
4. Multi-Faceted Bias Measurement: Successfully adapted RND, WEAT, and RNSB to the emoji setting, providing statistically grounded evidence of associative and sentiment biases across different model architectures.

4. Key Results

Coverage and Tokenization

• Static Models: Most legacy models (e.g., FastText, GloVe, BERT) support zero or very few skin-toned emojis. Only emoji-sw2v and emoji2vec showed substantial coverage.
• LLMs: All tested LLMs support the full spectrum of skin tones due to subword tokenization.
• Token Inequity: While Gemma and Qwen encode all modifiers as 1 token, and Llama uses 3, Mistral requires 2 tokens for most modifiers but 5 tokens specifically for the darkest skin tone. This creates a "computational penalty" for representing dark skin.

Semantic Consistency

• Alignment: Larger LLMs (e.g., Gemma-9B, Llama-8B) generally show better alignment between emojis and text than smaller variants.
• Drift:
  • Gemma: Shows semantic drift where dark skin tones are less aligned with their text descriptions than light tones.
  • Llama: Exhibits clustering where darker skin tones are consistently distanced from the "default" emoji in vector space.
  • Qwen: Shows a trend where alignment improves as skin tone darkens.

Bias Quantification

• Sentiment Bias (WEAT & RNSB):
  • Llama and Qwen: Consistently associate lighter skin tones with "Good" attributes and professional roles, while associating darker tones with "Bad" attributes.
  • Mistral: Displays the opposite bias, associating lighter skin tones with "Bad" attributes.
  • Qwen: Shows the highest RNSB (uneven distribution of negative sentiment), where default and dark skin tones bear a disproportionately higher share of negative sentiment compared to mid-tones.
• Societal Stereotypes (Caliskan WEAT):
  • Emoji2vec, Mistral, and Qwen systematically associate lighter skin tones with culturally positive concepts.
  • Gemma and Llama show the opposite pattern, associating darker skin tones with positive concepts (potentially a "counter-bias" or different training artifact).

5. Significance and Implications

• Beyond Support: The paper proves that "support" (the ability to process a token) does not equal "fairness." Models can process dark skin tones while simultaneously embedding them with negative sentiment or higher computational costs.
• Infrastructure Risk: As LLMs become the backbone of web interaction, these representational harms risk automating and scaling societal biases globally.
• Actionable Recommendations: The authors call for:
  • Tokenizer Normalization: Ensuring equal token counts for all skin tones to prevent computational discrimination.
  • Counterfactual Data Augmentation: Training models to ensure semantic consistency across skin tones.
  • Auditing: Developers must actively audit emoji embeddings for sentiment and associative biases before deployment.

In conclusion, the study highlights that while modern LLMs have solved the technical problem of representing skin tones, they have inherited or created new societal biases that require urgent mitigation to ensure genuine digital equity.