Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution

1. Problem Statement

Feature attribution methods (e.g., Integrated Gradients, LIME) are widely used to explain the predictions of language models by assigning importance scores to input tokens. However, these methods often produce inconsistent explanations for the same input and model.

• The Core Issue: This inconsistency stems from underlying biases within the attribution methods themselves, rather than just the model's decision-making process.
• The Risk: Users may either mistrust useful explanations or, more dangerously, trust unfaithful ones that reflect methodological artifacts rather than true model logic.
• Gap in Literature: While previous work has noted disagreement between methods, there is a lack of systematic frameworks to distinguish between lexical bias (preferring specific word types, e.g., punctuation vs. content words) and position bias (preferring tokens at specific locations, e.g., start or end of a sentence). Furthermore, it is unclear if newer, larger models (like ModernBERT) inherently suffer less from these biases.

2. Methodology

The authors propose a model- and method-agnostic evaluation framework to quantify bias using three specific metrics. They conduct experiments in two settings:

1. Controlled Artificial Setting: To isolate bias from task semantics.
2. Semi-Controlled Natural Setting: To observe bias in a realistic causal relation detection task.

A. Evaluation Framework (Three Metrics)

To differentiate sources of bias, the authors define:

1. Bias-cons (Inter-seed Consistency): Measures the consistency of bias distributions across different random seeds for the same model/method. High consistency implies the bias is a systematic property of the method/model, not random noise.
2. Bias-agg (Aggregate Bias): Measures the deviation of a method's aggregate distribution from a uniform baseline (representing "no bias"). This conflates model bias and method bias.
3. Bias-attr (Attribution Bias): Measures the pairwise divergence between a specific method and all other methods. High Bias-attr indicates a method is an outlier compared to the consensus.

B. Experimental Setup

• Models:
  • BERT: bert-base-uncased (110M parameters, fixed positional embeddings).
  • ModernBERT: ModernBERT-base (149M parameters, Rotary Positional Embeddings/RoPE).
• Attribution Methods: Six methods tested: PartSHAP, LIME, Vanilla Gradient (VanGrad), Grad × I, Integrated Gradient (IntGrad), and IntGrad × I.
• Datasets:
  • Artificial: Three datasets with random labels to ensure the model learns nothing meaningful:
    • noun-det-period: Nouns, determiners, periods.
    • period-comma: Only punctuation.
    • unique-punctuation: Unique set of punctuation marks.
  • Natural: A binary causal relation detection task using permuted sentence triples (positive: 3 causal sentences; negative: 3 non-causal sentences).

C. Faithfulness Sanity Check

The authors evaluate standard faithfulness metrics (Sufficiency and Comprehensiveness) to see if they can detect these biases. They hypothesize that in a controlled setting where the model learns nothing, these metrics should fail to distinguish biased methods.

3. Key Results

A. Artificial Data Findings

• Systematic Bias Exists: All methods exhibit systematic lexical and position biases, even when the model has no signal to learn (chance-level F1 scores).
• Trade-off Between Biases: There is a distinct trade-off between lexical and position bias. Models/methods that score high on one type of bias often score low on the other.
  • Example: Vanilla Gradient shows extreme position bias (heavily favoring the start of the input) but relatively lower lexical bias.
  • Example: Integrated Gradient is more balanced regarding position but shows strong lexical bias (preferring periods over commas).
• Model Comparison:
  • ModernBERT generally exhibits lower position bias than traditional BERT (likely due to Rotary Positional Embeddings).
  • However, ModernBERT does not consistently show lower lexical bias; in some cases, it exhibits higher lexical bias than BERT.
  • Conclusion: A "modern" or larger model does not guarantee reduced explanation bias.
• Outlier Detection: Methods with high Bias-attr (outliers) consistently correspond to high Bias-agg (most biased against the baseline). This suggests that methods diverging most from the consensus are likely the most unfaithful.
• Failure of Faithfulness Metrics: Standard metrics (Sufficiency/Comprehensiveness) failed to detect the high bias of Vanilla Gradient in this setting. They returned near-zero scores for all methods, correctly indicating the model learned nothing, but failing to differentiate between a "faithful" and "biased" explanation.

B. Natural Data Findings (Causal Detection)

• High Performance: Models achieved >0.98 F1, confirming they learned the task.
• Position Bias Persistence: BERT showed a tendency to attribute importance to the last sentence of the input triple, particularly for negative examples (where no causal signal exists).
• Lexical Preferences:
  • BERT (LIME): Focused on causal verbs (e.g., "produce," "increase").
  • ModernBERT (IntGrad): Selected a mix of verbs, punctuation, and topic words, sometimes producing "implausible" explanations (e.g., highlighting punctuation).
• Correlation: The relationship between Bias-attr and Bias-agg held true here as well: the most divergent methods were the most biased.

4. Key Contributions

1. Taxonomy of Bias: The paper formally structures attribution bias into Lexical (what) and Position (where) categories, demonstrating they are distinct and often inversely related.
2. Evaluation Framework: Introduces a three-metric framework (Bias-cons, Bias-agg, Bias-attr) to disentangle whether bias stems from the model, the random seed, or the attribution method itself.
3. Empirical Evidence on Modern Models: Challenges the assumption that newer, larger architectures (ModernBERT) are inherently more robust to explanation bias.
4. Critique of Faithfulness Metrics: Demonstrates that standard faithfulness metrics (Sufficiency/Comprehensiveness) are insufficient for detecting bias in models that do not rely on specific input features (or in low-resource settings).
5. Outlier Heuristic: Provides empirical evidence that attribution methods which deviate significantly from the consensus (high Bias-attr) are likely to be the most biased and unfaithful.

5. Significance and Implications

• For Practitioners: Users of XAI tools should not assume a single method provides the "truth." If a method's explanation is an outlier compared to others, it is likely biased.
• For Model Developers: Simply increasing model size or updating architecture (e.g., switching to RoPE) does not solve explanation bias; specific mitigation strategies for attribution methods are needed.
• For Research: The paper highlights the need for new evaluation metrics that can detect bias in "unfaithful" scenarios, as current metrics may give a false sense of security.
• Generalizability: The findings suggest that explanation bias is a product of the attribution method, not just the model, urging a shift in how we evaluate and trust post-hoc explanations in NLP.