Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis

Technical Summary: Relating Word Embedding Gender Biases to Gender Gaps

Problem Statement
Machine learning models for Natural Language Processing (NLP), particularly those utilizing word embeddings, are known to inherit biases from their training data, often reflecting societal stereotypes regarding race and gender. While recent research focuses on debiasing these models to improve accuracy and remove stereotypes, this paper posits an alternative perspective: these inherent biases may not merely be errors, but rather reflections of actual statistical disparities (gender gaps) within the cultures that produced the training text. The core problem addressed is whether the magnitude and direction of gender bias in word embeddings can be quantitatively correlated with real-world statistical gender gaps in education, politics, economics, and health across different cultures.

Methodology
The authors conducted a cross-cultural analysis using Twitter data from 2018, covering 99 countries and 51 U.S. regions (50 states and Washington, D.C.).

1. Data Collection and Preprocessing:

   • The dataset consisted of English-language tweets collected over 100 days in 2018.
   • Tweets were filtered to include only those with three or more words. URLs, usernames, hashtags, images, and emojis were replaced with tokens.
   • Corpora were constructed for each country and state. For cultures exceeding 10 million tweets, a sample of 10 million was used. The smallest corpus contained 98,000 tweets (Mauritius), and the largest 122 million (U.K.).
   • The authors noted these corpora are orders of magnitude smaller than those used in other tweet embedding studies (e.g., Li et al., 2017).

2. Embedding Construction:

   • The primary algorithm used was Word2Vec (skip-gram).
   • For comparative analysis, the authors also tested GloVe, Word2Vec CBOW, and FastText.
   • Parameters included a window size of 10, a minimum word frequency of 5, and 200-dimensional output vectors.

3. Bias Measurement:

   • The authors defined a "male-female axis" by averaging the vectors of male-specific words ($\vec{male}$) and female-specific words ($\vec{female}$).
   • Primary Metric: The average axis projection of a thematic neutral word set ($W$) onto the male-female axis was calculated using the formula:
     $$\text{avg}_{w \in W} \left( \frac{\vec{w} \cdot (\vec{female} - \vec{male})}{||\vec{female} - \vec{male}||^2} \right)$$
   • This projection serves as the measure of gender bias for specific thematic word sets (e.g., government, threat, childcare, intellect).

4. Correlation Analysis:

   • The authors correlated the bias scores of thematic word sets against 18 international gender gap statistics (from the Global Gender Gap Index) and 5 U.S.-based statistics (from the CDC, U.S. Census Bureau, and Represent Women).
   • Feature Selection: To prevent overfitting, the algorithm performed feature selection on 20% of the countries/states to down-select words within a thematic set, then computed the $R^2$ coefficient of determination against the full dataset. This process was repeated five times and averaged.

Key Results
The study found significant, selective correlations between linguistic biases and statistical gender gaps:

• International Findings:

  • Political Empowerment: The "government" word set showed a positive correlation with the Political Empowerment Gender Gap index. As political language exhibited a stronger female bias, the gender gap in political power decreased (i.e., women's influence increased).
  • Economic Participation: "Intellect" and "workplace" word sets positively correlated with economic participation statistics.
  • Health: "Illness" terms showed indirect correlations with health and survival statistics.
  • Overall Gap: The single word "pretty" demonstrated the strongest determination ($R^2$) against the overall gender gap index.
  • Random Baselines: Four randomly sampled word sets showed no significant correlation ($R^2 < 0.09$) with any gender gap, confirming that the results were not artifacts of the methodology.

• U.S. State Findings:

  • Pay Gap: The "threat" word set (e.g., dangerous, toxic, suspicious) showed a strong inverse correlation ($R^2 = 0.51$) with the gender pay gap. In states where threat words were more associated with females, the pay gap was larger (women earned fewer cents on the dollar).
  • Education and Workforce: "STEM-alum" words correlated with higher percentages of female math and CS degrees. "Criminal" behavior words negatively correlated with the proportion of female state legislators.
  • Exercise: Words describing "persistence and devotion" correlated with reduced gender gaps in exercise rates.

• Algorithm and Metric Comparison:

  • Among the four algorithms tested, Word2Vec (skip-gram) combined with the axis projection metric yielded the highest coefficients of determination for both direct and indirect correlations.
  • Valence and Dominance: Adjectives correlated with reduced gender gaps (e.g., higher wages, more female legislators) had significantly higher valence (positivity) and dominance scores than those correlated with increased gaps.

Key Contributions

1. Quantification of Bias-Gap Relationships: The paper provides a method to quantify gender bias in word embeddings and directly correlate it with statistical gender gaps across 99 countries and 51 U.S. regions.
2. Thematic Specificity: It demonstrates that gender biases are not monolithic; specific thematic word sets (e.g., government, threat) correlate with specific domains of gender disparity (e.g., political power, pay gap), while random word sets do not.
3. Social Media as a Cultural Proxy: The study validates that Twitter data, even with smaller corpus sizes and English-only constraints, can serve as a plausible source for computing cultural gender biases that align with real-world statistics.
4. Algorithmic Validation: It identifies Word2Vec with axis projection as the most effective configuration for detecting these specific correlations compared to GloVe, FastText, and alternative bias metrics.

Significance and Claims
The authors claim that their work supports three main assertions:

1. Cultural gender biases in language are associated with actual statistical gender gaps.
2. These biases can be characterized by the strength and direction of their correlation with specific gaps.
3. Themed word sets capture different dimensions of gender bias and gaps, allowing for nuanced analysis.

The paper concludes that these findings align with social theories suggesting that differences in implicit gender valuation (linguistic bias) manifest in differences in gender opportunities and status. However, the authors explicitly state that their analysis illustrates correlations, not causality. They caution that the interpretive power is limited and that the use of English-only tweets creates cultural blind spots, excluding populations without technology access or English proficiency. Future work is suggested to incorporate causal models and expand to other languages.