💬 NLP

GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations

This paper introduces GECOBench, a gender-controlled dataset and evaluation framework that demonstrates how fine-tuning pre-trained models like BERT, particularly by retraining embedding layers, significantly mitigates gender bias in the feature attributions generated by explainable AI methods.

Original authors: Rick Wilming, Artur Dox, Hjalmar Schulz, Marta Oliveira, Benedict Clark, Stefan Haufe

Published 2026-01-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rick Wilming, Artur Dox, Hjalmar Schulz, Marta Oliveira, Benedict Clark, Stefan Haufe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot named BERT. BERT has read millions of books, websites, and articles (like Wikipedia) to learn how to speak and understand us. Because it read so much, it also absorbed the hidden prejudices and stereotypes found in those old texts. For example, it might subconsciously think "doctor" goes with "he" and "nurse" goes with "she," even when those words aren't actually the reason for a decision.

Now, imagine you ask BERT to explain why it made a specific decision. It points to certain words and says, "I chose this because of these words!" This is called Explainable AI (XAI). The problem is: Is BERT telling the truth, or is it just pointing to the words that match its old stereotypes?

This paper introduces a new tool called GECOBench to test if these AI explanations are honest, specifically regarding gender.

The "Gender-Flip" Game (GECO Dataset)

To test BERT, the researchers created a special playground called GECO. Think of it as a game of "Mad Libs" but with a strict rule: Only change the pronouns.

They took a sentence like:

"She loves to spend time with her favorite cat."

Then, they created two identical twins of this sentence, changing only the gendered words:

Male version: "He loves to spend time with his favorite cat."
Non-binary version: "They love to spend time with their favorite cat."

Everything else—the cat, the loving, the time spent—stays exactly the same.

Why is this a game?
In this game, the only reason the answer changes from "Male" to "Female" is the pronoun. If an AI looks at the sentence and says, "I know this is about a female because of the word 'cat'," it is wrong. The "cat" is a red herring; it's a distraction. The only "truthful" clue is the pronoun.

Because the researchers built the sentences this way, they know the Ground Truth: The AI must point to the pronoun to be correct. If it points to the cat, the explanation is a lie (or at least, biased).

The "Truth Detector" (GECOBench)

The researchers used this dataset to build a testing framework called GECOBench. They fed these sentences to BERT and asked, "Which words made you decide this?"

They then compared BERT's answer against the "Ground Truth" (the pronoun). They used a scoring system called Mass Accuracy.

Perfect Score: The AI points only at the pronoun.
Low Score: The AI points at the pronoun and other unrelated words (like "cat" or "kitchen"), or misses the pronoun entirely.

What They Found

The researchers tested BERT in different "training modes" to see how to fix its bad explanations:

The "Frozen" BERT: They let BERT use its pre-trained brain without changing anything.
- Result: The explanations were messy. BERT kept pointing to stereotypical words (like "kitchen" or "car") instead of just the pronoun. It was biased.
The "Fine-Tuned" BERT: They retrained specific parts of BERT's brain on their new "Gender-Flip" sentences.
- Result: When they retrained the embedding layer (the part of the brain that understands the basic meaning of words), the explanations got much better. BERT finally learned to ignore the stereotypes and focus only on the pronoun.

The Big Takeaway:
Even if a model gets the right answer (e.g., correctly identifying "He" as male), its explanation might still be lying if it's relying on old biases. The paper shows that you can't just trust an AI's explanation; you have to check if it's pointing at the right clues.

The "Pattern" Baseline

The researchers also compared BERT to a simple, old-school math method called the Pattern Variant. This method doesn't have a "brain" full of biases; it just looks at the math of how words appear.

Surprise: The simple math method was actually better at finding the "truth" than the complex AI in many cases. This proves that the AI's complexity was getting in the way of honesty.

Summary

The Problem: AI models often give explanations that look good but are actually based on hidden biases (like gender stereotypes).
The Solution: A new dataset (GECO) where the only thing that changes is gender, creating a "truth" that the AI must follow.
The Result: By retraining the AI's understanding of words (embeddings), we can force it to give honest explanations that focus on the actual clues, not the stereotypes.

The paper concludes that this is a first step. It's like teaching a student to stop guessing based on stereotypes and start looking at the actual evidence. While this specific test is about gender, the method can be used to check if AI is being honest about any topic.

Technical Summary: GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations

Problem Statement

Large pre-trained language models (PLMs), such as BERT, are trained on vast corpora containing inherent societal biases, including gender stereotypes. While these biases are known to affect model weights and prediction behavior, it remains unclear to what extent they propagate into feature attributions generated by Explainable Artificial Intelligence (XAI) methods. Specifically, there is a lack of rigorous, data-centric frameworks to evaluate whether XAI methods correctly identify the features driving a prediction or if they inadvertently highlight biased, non-causal correlations present in the pre-training data. Current benchmarks often rely on "faithfulness" metrics or human-annotated rationales, which may not align with statistical associations or ground truth feature importance, making it difficult to objectively assess the correctness of explanations in the presence of bias.

Methodology

1. The GECO Dataset

The authors introduce GECO, a gender-controlled text dataset designed to provide ground truth feature attributions for gender classification tasks.

Construction: Sentences are sourced from Wikipedia pages corresponding to classic novels (via the Gutenberg project).
Manipulation: Each sentence is generated in three grammatical gender variants: Male ( $x^M$ $x^{M}$ ), Female ( $x^F$ $x^{F}$ ), and Non-binary ( $x^{NB}$ $x^{N B}$ ).
- Subject-only manipulation ( $D_S$ ): Only the grammatical subject pronouns are altered (e.g., "She" $\to$ "He"). This creates a more challenging task where the model must distinguish between subject and object genders.
- All-gender manipulation ( $D_A$ ): All gender-related words (pronouns, possessives, and gendered nouns like "Aunt" $\to$ "Uncle") are altered.
Ground Truth: By construction, the altered words are the only discriminative features. Therefore, the ground truth feature attribution map is binary: 1 for altered words, 0 for all others. This allows for the objective evaluation of whether an XAI method highlights the statistically associated features (Statistical Association Property - SAP).
Bias Verification: The authors verify that the dataset itself is unbiased using a co-occurrence metric, ensuring that deviations in model explanations stem from the model's pre-training biases rather than the data.

2. The GECOBench Framework

GECOBench is a quantitative evaluation framework applied to the GECO dataset to assess XAI performance.

Models: The framework evaluates the pre-trained BERT model under four distinct training paradigms to isolate the impact of different architectural layers:
1. BERT-C: Only the classification layer is re-trained; Embedding and Attention layers are fixed.
2. BERT-CE: Embedding and Classification layers are re-trained from scratch; Attention is fixed.
3. BERT-CEf: Embedding and Classification layers are fine-tuned; Attention is fixed.
4. BERT-CEfAf: Embedding, Attention, and Classification layers are all fine-tuned.
- Baselines: A zero-shot BERT model (BERT-ZS) and a vanilla one-layer attention model (OLA-CEA) trained from scratch on GECO (free of pre-training bias) are used for comparison.
XAI Methods Evaluated: The study applies various post-hoc attribution methods, including gradient-based approaches (Saliency, InputXGradient, DeepLift, Guided Backpropagation, Integrated Gradients) and surrogate methods (LIME, Kernel SHAP, Gradient SHAP).
Evaluation Metric: The Mass Accuracy (MA) metric is used to quantify correctness. MA measures the overlap between the normalized attribution scores and the ground truth binary mask, penalizing false positives (highlighting non-ground-truth words) while being robust to false negatives (missing some ground truth words).
Bias Quantification: The authors define Relative Mass Accuracy (RMA) to measure the deviation of explanation performance from a zero-shot baseline, quantifying "explanation bias."

Key Contributions

GECO Dataset: A novel, gender-controlled dataset with ground truth feature attributions at the word level, enabling the objective assessment of XAI correctness based on statistical association.
GECOBench: A rigorous benchmarking workflow to quantify how pre-training biases affect feature attributions and how fine-tuning strategies mitigate these biases.
Empirical Analysis of Bias in Explanations: The paper demonstrates that feature attribution methods are susceptible to pre-training biases and that the correctness of these explanations depends heavily on which layers of the model are fine-tuned.

Results

Explanation Performance (RQ1)

Dataset Difficulty: Performance on the subject-only dataset ( $D_S$ ) is generally lower than on the all-gender dataset ( $D_A$ ) across all methods, likely due to the reduced number of discriminative tokens.
Method Comparison: Integrated Gradients, LIME, and Gradient SHAP consistently outperform other methods and the random baseline.
Baseline Comparison: The Pattern Variant (a model-independent method based on input-target covariance) serves as an upper bound for performance. While no method reaches the Pattern Variant's score, Integrated Gradients and Gradient SHAP perform well in the transfer learning regime.
Unbiased Model: The OLA-CEA model (trained from scratch without pre-training) shows significantly higher explanation performance across all methods, confirming that pre-trained biases degrade explanation quality.

Bias Mitigation (RQ2)

Layer Dependency: There is a clear dependency between explanation performance and the number of fine-tuned layers.
Embedding Impact: Models where the embedding layer is re-trained or fine-tuned (BERT-CE, BERT-CEf) show the most significant improvement in explanation correctness compared to models where embeddings are fixed (BERT-C).
Residual Bias: Even when classification accuracy is held constant across different training regimes, models with fixed embeddings retain residual biases that negatively impact the correctness of feature attributions. Fine-tuning the embeddings is shown to be the most effective strategy for mitigating these biases.
Specific Observations: In some cases, XAI methods on biased models highlight irrelevant words (e.g., "Bella" instead of the pronoun "she") due to pre-training associations, whereas unbiased models correctly focus on the altered pronouns.

Significance and Claims

The paper claims that GECOBench provides the first principled, data-centric framework for evaluating the correctness of feature attributions in NLP, specifically addressing the tension between model-centric and data-centric explanations.

Bias Propagation: The study establishes that biases in pre-training corpora are not limited to prediction outputs but also permeate the explanations generated by XAI methods.
Correctness as a Bias Indicator: The authors argue that the data-centric correctness of feature attributions (specifically adherence to the SAP) is indicative of underlying model biases.
Mitigation Strategy: The results suggest that simply fine-tuning the classification head is insufficient for obtaining unbiased explanations; re-training or fine-tuning the embedding layers is necessary to mitigate the influence of pre-training biases on feature attributions.
Scope: The authors position this work as a first step toward systematic bias evaluation in XAI. They acknowledge limitations, such as the simplification of gender to pronoun alterations and the focus on univariate statistical associations, but emphasize that the framework successfully isolates and quantifies the impact of bias on explanation correctness in a controlled setting.

The paper concludes that caution is required when using XAI methods for model debugging or downstream tasks (like sentiment analysis) on pre-trained models, as biased attributions may highlight spurious gender-related tokens rather than semantically relevant features.