Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures

1. Problem Statement

Despite the state-of-the-art performance of Large Language Models (LLMs) in Machine Translation (MT), they exhibit systematic gender biases. This issue is particularly acute in translation tasks involving languages with different morphological gender systems (e.g., translating from gender-neutral English to gender-marked Italian or German).

• The Core Challenge: Models often fail to utilize explicit contextual gender cues (e.g., pronouns like he/she) and instead default to stereotypical associations or a masculine default.
• Limitations of Current Evaluation: Standard accuracy metrics (checking if the output gender matches the source) fail to distinguish between a model correctly using context and a model simply guessing the stereotypical gender. Furthermore, existing benchmarks do not quantify a model's "default" gender assumption when no explicit cue is present.
• Architectural Shift: While early MT systems used Encoder-Decoder architectures, current SOTA systems are predominantly Decoder-Only LLMs. It remains unclear whether these new architectures inherently handle gender disambiguation better or if they simply inherit and amplify biases from their massive, uncurated pretraining data.

2. Methodology

The authors extend a previous evaluation framework (Manna et al., 2025) to specifically target decoder-only models. The methodology involves three main components:

A. Models Evaluated

The study focuses on three decoder-only models of identical size (7B parameters, 32 layers) to isolate the effects of post-training:

1. Llama2: A general-purpose pretrained baseline.
2. TowerBase: Llama2 continued pretraining on monolingual and parallel MT corpora.
3. TowerInstruct: TowerBase further refined via instruction tuning on translation-specific tasks.
   Comparison: Results are compared against previously studied Encoder-Decoder baselines (OPUS-MT, NLLB-200, mBART).

B. Data and Prompting

• Dataset: The WinoMT dataset (Stanovsky et al., 2019), which contains sentences with ungendered profession nouns and gendered pronouns (e.g., "The cook... because he...").
• Neutral Set (Novel Contribution): The authors constructed a Neutral Set by replacing gendered pronouns (he/she) with neutral forms (they/them) and adjusting agreement. This allows for the measurement of Prior Bias (the model's default gender assumption in the absence of cues).
• Prompting: Zero-shot prompting was used to isolate model behavior, avoiding prompt engineering optimizations.

C. Evaluation Metrics

The framework combines three metrics:

1. Standard Gender Accuracy: Measures if the translated profession matches the source pronoun's gender.
2. Minimal Pair Accuracy (MPA): A stricter metric. A pair of sentences (differing only by he vs. she) is only counted as correct if the model produces the appropriate gendered translation for both cases. This ensures the model is actually using context, not just guessing stereotypes.
3. Prior Bias: A novel metric measuring the proportion of masculine vs. feminine forms generated on the Neutral Set.

D. Internal Analysis (Attention)

The authors analyzed self-attention patterns to understand how models process cues. They measured the attention weights allocated to gender pronouns when generating the target profession noun, specifically focusing on layers and heads where attention is strongest.

3. Key Contributions

1. Introduction of "Prior Bias": A new diagnostic measure that quantifies a model's inherent gender preference when explicit cues are removed.
2. Extension to Decoder-Only Architectures: The first comprehensive diagnostic evaluation of gender bias specifically for decoder-only LLMs in MT, comparing them against traditional encoder-decoder systems.
3. Diagnostic Framework: A combined approach using output-level metrics (MPA, Prior Bias) and internal attention analysis to distinguish between "ignoring context" and "defaulting to stereotypes."
4. Insight into Post-Training: A controlled study isolating the effects of continued pretraining vs. instruction tuning on bias mitigation.

4. Key Results

A. Decoder-Only vs. Encoder-Decoder

• No Inherent Advantage: Decoder-only models (even SOTA ones) do not outperform encoder-decoder systems on gender-specific metrics. In some cases, they perform worse.
• Masculine Default: All models exhibit a strong bias toward masculine forms, particularly in anti-stereotypical contexts (e.g., translating "housekeeper" as masculine when the pronoun is he, or failing to use feminine forms for male pronouns in stereotypical roles).

B. Impact of Post-Training

• Continued Pretraining (TowerBase): Had little to no benefit for gender disambiguation. In fact, TowerBase performed slightly worse than the base Llama2 on anti-stereotypical cases.
• Instruction Tuning (TowerInstruct): Showed significant improvements:
  • MPA: Increased from ~13% (Llama2) to 36.7%, indicating much better integration of contextual cues.
  • Feminine Accuracy: Rose sharply from 23.3% to 52.0%.
  • Prior Bias: The default masculine bias decreased from ~85% to 75.2% (a marked reduction, though still dominant).
  • Balance: TowerInstruct achieved a near-perfect split between stereotypically feminine and masculine professions in minimal pairs (50.4% / 49.6%), unlike the base models which heavily favored feminine stereotypes.

C. Attention Mechanism Insights

• Attention Patterns: Feminine pronouns trigger stronger and more localized attention than masculine ones, likely compensating for their lower frequency in training data.
• Timing of Improvement:
  • TowerBase showed increased attention to cues compared to Llama2, but this did not translate to better output (the model "saw" the cue but didn't "act" on it).
  • TowerInstruct was the only model where improved attention signals correlated with a reduction in Prior Bias and better output. This suggests instruction tuning helps the model leverage learned attention signals to override the masculine default.

5. Significance and Conclusion

• Diagnosis over Description: The paper moves beyond simply stating "models are biased" to diagnosing why (ignoring cues vs. defaulting to priors) and where (specific attention heads/layers).
• Instruction Tuning as a Mitigation Strategy: The study provides evidence that instruction tuning is a promising, scalable method for reducing gender bias in MT, potentially more effective than continued pretraining alone. It suggests that teaching models to follow instructions helps them override statistical priors.
• Limitations: The study is currently limited to binary gender (masculine/feminine) due to dataset constraints, and the attention analysis is correlational rather than causal.
• Future Directions: The authors call for evaluation frameworks that include non-binary gender and causal interpretability techniques to further understand and mitigate bias in the era of LLMs.

Summary: While decoder-only LLMs have revolutionized MT quality, they do not inherently solve gender bias. However, instruction tuning serves as a critical intervention, significantly improving contextual awareness and reducing the default masculine bias that plagues these models.