Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

Technical Summary: Fairness Pruning

Problem Statement
Large Language Models (LLMs) inherit and amplify demographic biases present in their pretraining corpora. Current mitigation strategies operate at three distinct lifecycle phases: pretraining data filtering, post-training fine-tuning/alignment, and post-processing representation editing. The authors argue that these approaches share a structural deficit: they intervene on all model weights simultaneously or modify embeddings without distinguishing between weights encoding bias and those supporting general capabilities. This lack of specificity risks degrading general model performance or causing catastrophic forgetting. While mechanistic interpretability (e.g., Sparse Autoencoders) offers precision, it requires significant computational resources for additional training, creating a gap between mass intervention and precise, affordable dissection of bias.

Methodology: Fairness Pruning
The paper proposes "Fairness Pruning," a lightweight, structural intervention method designed to locate and mitigate demographic bias without additional training. The method is specifically tailored for GLU-based MLP architectures (e.g., Llama, Salamandra) and operates through a four-phase pipeline:

1. Contrastive Prompt Construction: The method utilizes minimally contrastive prompt pairs where the only difference is a single demographic attribute (e.g., "man" vs. "woman"). To ensure valid activation comparison, both prompts in a pair must tokenize to the exact same number of tokens. The authors released two public datasets (English and Spanish) covering five categories: Age, Gender, Physical Appearance, Race/Ethnicity, and Religion.
2. Differential Activation Capture: During inference, the method captures activations at the input of the down_proj layer in MLP blocks. This specific point is chosen because, in SwiGLU architectures, the effective contribution of a neuron is the product of the gate_proj and up_proj outputs, which is then projected by down_proj.
3. Neuron Scoring:
   • BiasScore: Calculated as the absolute difference between the mean activations of the two prompts in a pair, averaged over all pairs in a category. This identifies neurons that react differentially to demographic changes.
   • FairnessPruningScore: A composite metric combining BiasScore with structural importance (estimated via Peak-to-Peak Magnitude, PPM). However, in the exploratory experiments (removing ≤40 neurons), the authors found that BiasScore alone was sufficient, as the intervention scale was too small to require penalizing structural importance.
4. Selective Zeroing: The identified neurons are "zeroed out" by setting the corresponding rows in gate_proj and up_proj weights, and the corresponding column in down_proj weights, to zero. This symmetric operation ensures the neuron is removed from the residual stream.

Key Contributions

1. A Training-Free Localization Method: A method to locate demographic bias in LLMs using differential activations on contrastive prompts, runnable on consumer hardware without additional training.
2. Public Datasets: The release of two datasets of contrastive prompt pairs in English and Spanish, aligned with the BBQ and EsBBQ benchmarks.
3. Empirical Evidence of Dissociable Circuits: Demonstration that demographic attribute processing and general model capabilities operate on sufficiently dissociable circuits, allowing targeted intervention without appreciable degradation of reasoning or knowledge.
4. Discovery of Bidirectional Destabilization: Evidence that zeroing neurons selected by an unsigned differential activation score produces non-monotonic, sign-inverting effects on aggregate bias. This indicates that candidate sets mix neurons that amplify stereotypes with those that suppress them, and the net effect depends on which sign dominates.

Results

• Localization: Bias signals are robustly localized in the final layers of the MLP blocks. In Salamandra-2B, up to 94% of the most biased neurons for a category were found in the final layer. The spatial structure of these neurons is scattered across the layer width, not contiguous.
• Circuit Specificity: There is low overlap between circuits for different demographic categories (e.g., Gender circuits share very few neurons with others), suggesting functional specialization.
• Intervention Effects:
  • Bias: Zeroing identified neurons did not produce a flat reduction in bias. Instead, it caused "bidirectional bias destabilization." In some cases, bias was reduced; in others, it was amplified or inverted (e.g., shifting from stereotypical to anti-stereotypical). This confirms that the selected neurons act as regulators with opposing effects, and zeroing them blindly disrupts the modulation mechanism.
  • Capabilities: The intervention was extremely surgical. Zeroing up to 40 neurons in Llama-3.2-1B (less than 0.031% of total MLP width) resulted in a mean retention of 99.49% in reasoning and general knowledge benchmarks (WikiText, MMLU, ARC-Challenge, HellaSwag).
• Qualitative Observations: Generated text examples showed that the same set of zeroed neurons could reduce bias in one context while amplifying it in another, or leave factual knowledge (e.g., "Capital of France") completely unchanged.

Significance and Claims
The paper claims to establish the methodological foundations for transitioning from "blind zeroing" toward "directional behavior modulation."

• Dissociability: The primary finding is that bias processing and general capabilities are dissociable, validating the hypothesis that bias is concentrated in identifiable neurons rather than diffusely spread.
• Limitations of Unsigned Metrics: The work highlights that using unsigned differential scores (magnitude only) leads to unpredictable outcomes because it conflates amplifiers and regulators.
• Future Direction: The authors argue that the next step is not simply removing more neurons, but evolving the dataset to be asymmetric (knowing a priori which prompt is biased). This would allow for a signed BiasScore, enabling "activation steering" (multiplicative scaling) rather than zeroing, to push the model toward neutrality without gradient updates.

The study is modest in its scope, limited to models under 4 billion parameters and focusing on explicit demographic attributes via multiple-choice benchmarks. It explicitly excludes attention mechanisms and open-ended generation dynamics from its current validation, noting that the observed destabilization dynamics may not generalize directly to those settings without further study.