💬 NLP

ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation

This paper introduces the Contextual Gender Annotation (ConGA) framework, a linguistically grounded guideline for word-level gender annotation that addresses gender bias in machine translation from English to Italian by creating a gold-standard dataset to evaluate and improve the performance of current MT systems.

Original authors: Argentina Anna Rescigno, Eva Vanmassenhove, Johanna Monti

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Argentina Anna Rescigno, Eva Vanmassenhove, Johanna Monti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a translator trying to explain a story from English to Italian.

In English, the language is a bit like a chameleon when it comes to gender. You can say, "The doctor is here," and the word "doctor" doesn't tell you if the person is a man or a woman. It's neutral. You can also say, "They are happy," and "they" could be anyone.

But in Italian, the language is like a strict dress code. You must pick a gender. You can't just say "doctor"; you have to say il dottore (male doctor) or la dottoressa (female doctor). You can't just say "happy"; you have to choose contento (male happy) or contenta (female happy).

The Problem: The "Default" Setting

The problem is that when computers (Machine Translation systems) try to translate from English to Italian, they often get stuck. Since the English source doesn't give them a clue, the computer panics and picks the "safe" option: the male version.

It's like a waiter who, when asked "What would you like to drink?" and the customer says "Something neutral," automatically pours a beer because "most people drink beer," even if the customer might have wanted juice. Over time, this makes the world in the computer's mind look like a place where everyone is a man, reinforcing old stereotypes (like thinking all nurses are women and all engineers are men).

The Solution: ConGA (The Detective's Notebook)

The authors of this paper created a new set of rules called ConGA (Contextual Gender Annotation). Think of ConGA as a detective's notebook for translators.

Instead of just guessing, the researchers went through thousands of sentences and put "sticky notes" on the words to mark exactly what the gender situation was.

The English Side (The Clues): They looked at the English sentence and asked: "Do we know the gender?"
- M (Masculine): Yes, it's a man.
- F (Feminine): Yes, it's a woman.
- A (Ambiguous): We don't know. It could be either. (This is the tricky part).
The Italian Side (The Action): They looked at the Italian translation and asked: "Did the computer make a choice?"
- If the English was Ambiguous (A), but the Italian translation forced a male or female ending, the detective notebook flags it.
- If the computer guessed "Male" 9 times out of 10 when it didn't know, the notebook records that as a bias.

The Experiment: Testing the Computers

The researchers took two powerful AI translators (one called TowerLLM and another called mBART) and asked them to translate these sentences. Then, they used their "detective notebook" to grade the work.

What they found:

The "Male Default" is real: When the AI didn't know the gender, it almost always guessed "Male." It was like a coin that was weighted on one side.
Missing the Females: The AI was much better at getting the "Male" guesses right when it did know the gender, but it often missed the "Female" clues entirely, turning a female nurse into a male nurse by accident.
The Bias Score: They calculated a score (like a report card) showing that while the AI is getting smarter, it still has a heavy "male bias" habit that it can't break.

Why This Matters

This isn't just about grammar; it's about fairness.

If AI systems keep translating "The doctor" as "He" and "The nurse" as "She" (even when the English doesn't say so), they are teaching the world that men are the default humans and women are the exceptions.

ConGA is the tool that helps us catch this. It gives us a way to measure exactly how biased a computer is, so we can fix it. It's like putting a speed camera on a car that always speeds; once we can see the speed, we can finally build a system that drives fairly.

The Takeaway

This paper gives us a rulebook and a scorecard to stop computers from assuming everyone is a man just because they don't know better. By teaching the AI to recognize when it doesn't know the gender, and to leave it neutral (or ask for help) instead of guessing "Male," we can build a future where technology reflects the real, diverse world we live in.

1. Problem Statement

The paper addresses the persistent challenge of gender bias in Machine Translation (MT) and Large Language Models (LLMs), specifically in translation pairs moving from gender-neutral languages (e.g., English) to morphologically gendered languages (e.g., Italian).

Structural Asymmetry: English often omits grammatical gender (e.g., "I am a nurse" or "I am happy"), whereas Italian requires explicit gender agreement across nouns, adjectives, verbs, and pronouns.
The Bias: When the source text is ambiguous, MT systems statistically default to the masculine form, reinforcing societal stereotypes and reducing translation accuracy.
Evaluation Gap: Existing evaluation methods lack a unified framework to quantify this bias. Current approaches often rely on simple counts or benchmark datasets that lack linguistic diversity, failing to distinguish between necessary grammatical agreement and unjustified gender introduction.

2. Methodology: The ConGA Framework

The authors propose ConGA (Contextual Gender Annotation), a linguistically grounded framework designed to annotate gender at the word level to enable precise bias quantification.

A. Annotation Scheme

The framework uses a tripartite tagging system that distinguishes between semantic gender in the source and grammatical realization in the target:

Source (English):
- <M>: Masculine (explicit or strongly implied).
- <F>: Feminine (explicit or strongly implied).
- <A>: Ambiguous (gender cannot be inferred from context, e.g., "you," "parent," or neutral roles).
Target (Italian):
- <M> / <F>: Applied only to elements where gender is grammatically realized (e.g., adjectives, past participles, articles).
- No <A> tag in Italian: Ambiguous or unmarked forms are left untagged. This is crucial; if a model translates an ambiguous English source into a specific Italian gender, it is a "gender introduction" event that can be measured.
Entity Tracking: Tags are enriched with numerical identifiers (e.g., <A1>, <F1>) to track referents across sentences and ensure alignment between source and target entities.

B. Experimental Setup

Dataset: The authors utilized the gENder-IT dataset (derived from MuST-SHE), a bilingual English-Italian corpus.
Models Evaluated:
1. TowerLLM: A 7B instruction-following model (TowerInstruct).
2. mBART: A standard multilingual BART model.
Process:
1. Generation: Translations were generated using a low temperature (0.2) to ensure deterministic, semantically consistent outputs for reliable alignment.
2. Annotation: Human linguists manually annotated the English source and Italian target using the INCEpTION tool, applying the ConGA guidelines.
3. Alignment: The English annotations served as the Gold Standard. The Italian outputs were treated as model predictions.

C. Evaluation Metrics

The study employs standard classification metrics calculated at the entity level:

Precision: Ratio of correctly generated gendered forms in the output vs. total gendered forms produced by the model.
Recall: Ratio of correctly generated gendered forms vs. total gendered forms in the source.
Bias Detection: Specifically analyzes Mismatch Bias cases where an ambiguous source entity (<A>) is translated into a specific gender (<M> or <F>).
- Error: Mismatch between explicit source and target (e.g., Source <F> $\to$ Target <M>).
- Bias: Unjustified gender introduction (e.g., Source <A> $\to$ Target <M>).

3. Key Contributions

ConGA Framework: A novel, linguistically rigorous annotation guideline that separates semantic gender cues from grammatical realization, allowing for the isolation of bias from necessary grammatical agreement.
Gold-Standard Resource: The creation of a manually annotated, fine-grained parallel corpus (gENder-IT extended) that serves as a benchmark for evaluating gender bias in EN-IT translation.
Quantitative Bias Metric: A methodology to calculate bias scores by measuring the asymmetry in how models resolve ambiguous entities (A $\to$ M vs. A $\to$ F).
Integration of Linguistics and NLP: The work bridges the gap between linguistic theory (agreement rules, pro-drop languages) and computational evaluation (Precision/Recall/F1).

4. Results

The evaluation of TowerLLM and mBART against the gold standard revealed significant systemic biases:

Masculine Overuse: Both models exhibited a strong tendency to default to masculine forms when the source was ambiguous.
- Bias A $\to$ M: TowerLLM produced 215 masculine resolutions for ambiguous inputs; mBART produced 221.
- Bias A $\to$ F: TowerLLM produced only 35 feminine resolutions; mBART produced 29.
- Conclusion: The ratio of masculine to feminine resolutions for ambiguous inputs is heavily skewed (approx. 6:1), confirming a "male-default" heuristic.
Performance Metrics:
- Precision: TowerLLM showed slightly higher precision (31.8% for Male, 49.5% for Female) compared to mBART, indicating fewer erroneous gender flips.
- Recall: Both models struggled with recall, particularly for feminine forms, often failing to reproduce gendered forms present in the source or introducing them where none existed.
- F1-Score: The overall F1-scores were low (ranging from ~35% to 41%), highlighting that current systems are not yet reliable for gender-aware translation.
Error Types: While explicit errors (F $\to$ M) occurred, the dominant issue was Bias (A $\to$ M), where the model actively chose a gender despite the lack of source information.

5. Significance and Future Implications

Benchmarking: ConGA provides a scalable, reproducible benchmark for cross-system comparison, moving beyond simple BLEU scores to assess social fairness.
Mitigation Strategy: The annotated data can be used for Direct Preference Optimization (DPO) or fine-tuning to teach models to recognize when not to introduce gender (i.e., maintaining neutrality in the target language where the source is ambiguous).
Theoretical Insight: The study confirms that gender bias in MT is not merely a data imbalance issue but a structural heuristic where models actively apply a "male-default" when context is missing.
Future Work: The authors suggest exploring the impact of generation temperature on gender resolution and expanding the framework to other language pairs and non-binary identities.

In summary, ConGA offers a critical methodological shift from merely detecting gender errors to quantifying unjustified gender introduction, providing a robust tool for developing more equitable and linguistically aware NLP systems.