Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

Technical Summary: Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

Problem Statement
Large Language Models (LLMs) are increasingly deployed in socially sensitive contexts, yet they continue to encode and propagate gender biases. While Chain-of-Thought (CoT) prompting has been proposed as a mechanism to mitigate such biases by encouraging step-by-step reasoning, existing evaluations rely primarily on benchmark performance metrics. These metrics often fail to distinguish between genuine reductions in internal bias mechanisms and superficial alignment or benchmark compliance. Consequently, it remains unclear whether CoT prompting fundamentally alters how models encode, process, and manifest gender bias, or if observed improvements are merely artifacts of dataset familiarity and memorization.

Methodology
The authors present a systematic study combining benchmark-based evaluation, mechanistic interpretability, and reasoning chain failure analysis. The experimental setup involves:

1. Datasets and Models: The study evaluates five LLMs (Qwen2.5-7B, Qwen2.5-32B, QwQ, Llama3-8B, and Mistral-7B) across four gender-bias benchmarks: BBQ, StereoSet, CrowS-Pairs, and SocioEconomicQA. Prompts are formatted as multiple-choice questions with stereotypical, anti-stereotypical, and "unknown" (abstention) options.
2. Conditions: Models are tested under two conditions: Standard prompting and Zero-shot CoT prompting (appending "Let's think step by step").
3. Mechanistic Interpretability:
   • Stereotype Attention Score (SAS): Adapted from prior work, this metric quantifies the attention weight directed toward stereotypical versus anti-stereotypical tokens across specific attention heads. The authors identify clusters of attention heads where bias is concentrated.
   • Hidden State Probing: Linear probes (2-layer MLPs) are trained on hidden states extracted from reasoning chains to predict the model's selected answer type (stereotype, anti-stereotype, or unknown). This assesses whether gender bias information remains encoded in internal representations even when the output appears unbiased.
4. Reasoning Chain Analysis: A taxonomy of seven reasoning behaviors (Reasoning Correctness, Abstention, Dissociation, Task Hacking, Prompt Violation, Authority, and Bias) was developed. Three authors annotated a sample of 257 chains, and binary classifiers were trained to analyze 27,308 reasoning chains across all models and datasets.

Key Contributions

• Systematic Evaluation: The paper provides a comprehensive evaluation of CoT's effectiveness for gender bias mitigation across multiple models and datasets, challenging the assumption that CoT consistently reduces bias.
• Mechanistic Insight: It adapts attention head analysis and hidden state probing to the specific context of gender bias, revealing that bias persists in internal representations even when CoT is used.
• Reasoning Failure Taxonomy: The authors develop and apply a novel taxonomy to categorize reasoning chain behaviors, distinguishing between genuine reasoning and behaviors like "task hacking" or "dissociation" that lead to biased outputs.

Results

• Benchmark Performance: CoT prompting does not consistently reduce the bias gap (Diff-Bias). While it improved abstention rates for some models (Llama-8B, Mistral-7B), it decreased performance or increased bias scores for others (Qwen family, QwQ). The effect is highly dependent on model architecture, size, and dataset.
• Attention Mechanisms: CoT reduces the global Stereotype Attention Score (SAS) by balancing attention rather than suppressing biased heads. However, specific clusters of attention heads (e.g., around layers 8–17 for Qwen-7B) remain decisive in answer selection, and their activity is not consistently suppressed by CoT.
• Hidden Representations: Probing experiments reveal that gender bias information remains encoded in hidden states across all layers, regardless of whether CoT is used. High probe fidelity indicates that the model's internal state still distinguishes between biased and unbiased paths, suggesting CoT offers only superficial mitigation.
• Reasoning Behaviors: Analysis of reasoning chains shows that "Reasoning Correctness" does not guarantee unbiased outputs. Biased answers often co-occur with behaviors like "Dissociation" (inconsistent reasoning), "Task Hacking" (exploiting prompt grammar), and appeals to "Authority."
• Data Contamination: The study identifies evidence of dataset contamination. Models exhibit higher abstention rates and lower bias on familiar datasets (like BBQ) but struggle on unseen data (SocioEconomicQA). In some cases, reasoning chains contain verbatim text from the training data, suggesting that apparent bias mitigation stems from memorization rather than genuine reasoning.

Significance and Claims
The paper argues that CoT prompting produces a superficial behavioral correction rather than a fundamental change in how LLMs encode and manifest gender bias. The authors claim that:

1. Internal Bias Persists: Despite CoT-induced changes in output, gender bias remains embedded in hidden representations and specific attention head clusters.
2. Memorization vs. Reasoning: Improvements in bias metrics are often driven by dataset familiarity and memorization (data contamination) rather than a model's ability to reason about social biases.
3. Limitations of CoT: CoT is insufficient for mitigating social bias because it relies on statistical patterns and struggles to formalize exceptions or handle the implicit, non-formalizable nature of social bias.
4. Need for Deeper Mitigation: Genuine bias mitigation requires strategies that target the implicit representations underlying model behavior, rather than relying solely on prompt-based reasoning strategies that may encourage "overthinking" or evasive responses.

The authors conclude that while CoT can improve reasoning correctness, it does not reliably reduce gender bias and may even exacerbate it in certain contexts by inducing complex but flawed reasoning patterns.