GKnow: Measuring the Entanglement of Gender Bias and Factual Gender

Technical Summary: GKnow: Measuring the Entanglement of Gender Bias and Factual Gender

Problem Statement
Recent mechanistic interpretability research has focused on identifying and mitigating gender bias in Large Language Models (LLMs) by analyzing specific components responsible for biased outputs. However, current approaches suffer from two primary limitations: (i) they often target very specific tasks, such as gendered pronoun prediction, and (ii) they fail to distinguish between factual gender (the correct semantic association between a noun and its gender, e.g., "woman" $\to$ "she") and stereotypical gender (outputs based on social stereotypes, e.g., "nurse" $\to$ "she").

This lack of distinction leads to a critical side effect in debiasing efforts: the degradation of a model's factual gender knowledge. Existing work suggests that gender bias and factual gender signals are entangled within the model's internal representations, yet the mechanistic story of this entanglement at the circuit and neuron levels remains largely unexplained. Consequently, simple debiasing techniques like neuron ablation may inadvertently remove the model's ability to correctly predict factual gender while claiming success on bias benchmarks.

Methodology
To address these gaps, the authors propose a multi-faceted approach combining dataset curation, circuit analysis, and neuron ablation experiments on two decoder-only transformer models: Llama-3.1-8b and Olmo-7b.

1. Dataset Curation (GKnow):
   The authors introduce GKnow, a benchmark designed to assess gender knowledge and bias across diverse prediction types. GKnow categorizes prompts by the assumption (subject) and the expected output. The dataset includes:

   • Factual subsets: Based on lexically gendered nouns (e.g., "woman"), names, and explicit gender indicators.
   • Stereotypical subsets: Based on stereotypically gendered occupations (e.g., "nurse") or traits (e.g., "nurturing").
   • Prediction types: Gendered pronoun prediction, gender identification, and name prediction.
     GKnow allows for the separation of tasks into "stereotypical" (bias-prone) and "factual" (knowledge-based) categories to measure entanglement.

2. Circuit Analysis:
   Using EAP-IG (Edge Attribution Patching with Integrated Gradients), the authors identify minimal faithful circuits responsible for gendered predictions. They analyze:

   • Intersection over Union (IoU): Measuring the overlap of edges and nodes between circuits for different tasks.
   • Cross-task Faithfulness: Evaluating whether a circuit identified for one task (e.g., stereotypical prediction) can faithfully reproduce the behavior of a different task (e.g., factual prediction).

3. Neuron-Level Analysis and Ablation:
   The authors employ Integrated Gradients to identify task-relevant neurons, specifically those associated with stereotypical predictions (as these are the typical targets for debiasing). They perform neuron ablation (setting activations to zero) on the top 10 and 50 identified neurons.

   • Evaluation: The impact of ablation is measured on GKnow, StereoSet, and DiFair (a benchmark for disentangled assessment of gender knowledge and bias).
   • Metrics: The study tracks $P_{exp}$ (probability of expected token), $P_{opp}$ (opposite gender token), $P_{other}$, and the probability gap ($\Delta$) between masculine and feminine outputs.

Key Results

• Circuit-Level Entanglement:

  • Circuits responsible for stereotypical and factual gender predictions show high overlap and low separation.
  • Cross-task faithfulness is high: circuits identified for stereotypical tasks can recover the performance of factual tasks (and vice versa). For instance, applying the gender_prediction_based_on_stereo circuit to the gender_prediction_based_on_pronoun subset achieved a faithfulness of 1.0.
  • This indicates that factual and stereotypical gender are severely entangled at the circuit level.

• Neuron-Level Entanglement and Ablation Effects:

  • Ablating neurons identified via stereotypical prompts causes a significant decrease in the model's ability to predict factual gender.
  • In GKnow Factual subsets, ablation of top neurons led to sharp decreases in $P_{exp}$ (e.g., dropping from 91.49% to 79.86% in Llama for 50 neurons) and increases in $P_{opp}$ and $P_{other}$.
  • Crucially, while ablation often improves metrics on stereotypical benchmarks (e.g., reducing $P_{exp}$ for stereotypical predictions in StereoSet), these "positive" results mask a simultaneous collapse in factual gender competence.
  • The probability gap ($\Delta$) between masculine and feminine outputs decreases significantly, indicating a loss of confidence in gendered predictions across both factual and stereotypical subsets.

• Human Interpretability:

  • Using the Logit Lens technique, the authors found that shared neurons across different gender tasks are human-interpretable. For example, specific neurons in Olmo and Llama were found to suppress or promote specific gendered tokens (e.g., pronouns like "she" or names like "Mary") across both factual and stereotypical contexts.

Key Contributions

1. GKnow Benchmark: The curation of GKnow, which fills a gap in available resources for evaluating the entanglement of gender bias and factual gender knowledge across various task types in English.
2. Circuit-Level Evidence: The demonstration, via EAP-IG, that stereotypical and factual gender signals are entangled within the same computational circuits, challenging the assumption that they can be easily separated.
3. Neuron Ablation Analysis: The empirical finding that simple neuron ablation, while potentially effective at reducing stereotypical bias on specific benchmarks, severely compromises factual gender knowledge. This suggests that ablation is an unreliable standalone method for debiasing.

Significance and Claims
The paper argues that the entanglement of gender bias and factual gender is a fundamental property of current LLM architectures, observable at both the circuit and neuron levels. The authors claim that:

• Current debiasing methods are flawed: Techniques like neuron ablation, which target "bias neurons," inadvertently destroy factual knowledge because the underlying mechanisms are shared.
• Evaluation benchmarks are insufficient: Benchmarks that do not account for factual gender (like StereoSet) can hide the degradation of linguistic competence caused by debiasing attempts.
• Future work requires robust evaluation: There is an urgent need for gender bias evaluation datasets that explicitly measure the preservation of factual gender knowledge alongside bias reduction.

The authors conclude with a caution against "unfiltered neuron ablation" as the sole method for gender debiasing, emphasizing that without robust benchmarks like GKnow, the field risks improving bias metrics at the cost of basic language understanding. The paper acknowledges limitations, noting that findings are currently specific to English and relatively small models, and that gender bias may manifest in forms not covered by their occupational and adjective-based stereotypes.