💻 computer science

FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation

The paper proposes FairQE, a multi-agent framework that effectively mitigates gender bias in machine translation quality estimation by dynamically combining conventional scores with LLM-based reasoning to ensure fairer evaluation without compromising overall accuracy.

Original authors: Jinhee Jang, Juhwan Choi, Dongjin Lee, Seunguk Yu, Youngbin Kim

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinhee Jang, Juhwan Choi, Dongjin Lee, Seunguk Yu, Youngbin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very strict, highly trained Judge whose job is to grade machine translations. This Judge is supposed to be fair, objective, and accurate. However, researchers discovered that this Judge has a hidden flaw: it has a subconscious bias toward men.

If the Judge sees a sentence where the gender isn't clear (like "The doctor arrived"), it tends to give higher scores to translations that use "he" and lower scores to "she," even though the original sentence didn't specify. If the original sentence clearly says "She is a doctor," the Judge might still prefer the translation "He is a doctor" because it's used to thinking that way.

This paper introduces FairQE, a new system designed to fix this Judge without firing them. Think of FairQE not as a replacement, but as a team of specialized consultants working alongside the Judge to ensure a fair verdict.

Here is how the "FairQE Team" works, broken down into simple steps:

1. The Detective (Cue Detection)

First, a Detective agent scans the original sentence and the translation.

The Job: The Detective looks for "gender clues." Is the sentence about a "doctor" (ambiguous)? Or is it explicitly about "her" (explicit)?
The Analogy: Imagine the Detective is holding a magnifying glass. If they see a clue like "she," they flag it as "Explicit." If they see "doctor" with no pronoun, they flag it as "Ambiguous." If there are no clues at all, they say, "Nothing to see here," and the team skips the next steps to save time.

2. The Chameleon (Variant Generation)

Once the Detective finds a clue, a Chameleon agent steps in.

The Job: The Chameleon creates "what-if" versions of the translation.
- If the original was "He is a doctor," the Chameleon creates a version with "She is a doctor" and a neutral version.
- If the original was "The doctor is here" (ambiguous), the Chameleon creates versions with "He," "She," and "They."
The Analogy: Think of this like a costume shop. The Chameleon puts the same actor (the sentence) into different outfits (gendered versions) to see how the Judge reacts to each one.

3. The Two-Track Review (Dual-Stream Estimation)

Now, the team looks at the scores from two different angles:

Track A (The Veteran): This is the original Judge (the traditional AI model). It gives a score based on how fluent and grammatically correct the sentence sounds. It's fast and good at general quality, but it has that gender bias.
Track B (The Ethicist): This is a Large Language Model (LLM) acting as a Fairness Consultant. It looks at the "costume shop" versions created by the Chameleon.
- If the source was ambiguous: The Consultant asks, "Did the Veteran give the 'He' version a higher score just because it's 'He'? If so, that's unfair!"
- If the source was explicit: The Consultant asks, "Did the Veteran give the 'He' version a higher score even though the source said 'She'? That's a mistake!"

4. The Smart Mixer (Dynamic Aggregation)

This is the magic sauce. The system doesn't just average the two scores. It uses a Smart Mixer that decides how much to listen to the Fairness Consultant based on how "biased" the situation looks.

The Analogy: Imagine a volume knob.
- If the sentence is neutral and the Veteran Judge is doing a great job, the knob is turned down on the Consultant. The Veteran's score counts for almost everything.
- If the system detects a strong gender bias (e.g., the Veteran is wildly unfair to the "She" version), the knob is turned up. The Consultant's reasoning takes over, and the final score is adjusted to be fair.

Why is this a big deal?

Before this, fixing bias usually meant retraining the whole Judge from scratch, which is expensive and might make them worse at their actual job (grading translation quality).

FairQE is like a "plug-and-play" fairness filter.

It keeps the Judge's ability to spot bad grammar and fluency issues.
It adds a safety net that catches gender bias.
The Result: The paper shows that this team produces scores that are fairer (no longer favoring men by default) but are just as accurate (or even better) at judging the overall quality of the translation.

In a Nutshell

FairQE is a team of referees where one is the expert on grammar, and the other is the expert on fairness. They talk to each other, and if the grammar expert makes a biased mistake, the fairness expert steps in to correct the score, ensuring that a translation is judged on its quality, not on whether it uses "he" or "she."

1. Problem Statement

Quality Estimation (QE) aims to assess machine translation (MT) quality without reference translations. However, recent studies reveal that existing QE models exhibit systematic gender bias:

Gender-Ambiguous Scenarios: When the source text does not specify gender (e.g., "The doctor arrived"), QE models systematically assign higher scores to masculine translations (e.g., "El doctor") over feminine or neutral ones, reinforcing stereotypes.
Gender-Explicit Scenarios: When the source explicitly requires a specific gender (e.g., "She is a doctor"), QE models may still favor masculine translations or penalize correct feminine forms, a phenomenon known as preference inversion.

These biases compromise the fairness of MT evaluation and can negatively impact downstream tasks like model selection and data filtering. Current solutions often lack concrete methodological frameworks to mitigate these biases without sacrificing general evaluation accuracy.

2. Methodology: The FairQE Framework

The authors propose FairQE, a multi-agent, fairness-aware framework that integrates traditional QE models with Large Language Model (LLM) reasoning. The framework operates in four sequential stages using five agents (four LLM-based, one conventional QE model):

A. Gender Cue Detection (Agent_cue)

Function: Analyzes source-target pairs to identify gender-related linguistic cues.
Taxonomy: Uses a 12-category taxonomy distinguishing between Gender-Ambiguous (e.g., neutral nouns, generic plurals) and Gender-Explicit (e.g., pronouns, gendered kinship terms) cues.
Output: A set of detected cue pairs linking source spans to target spans. If no cues are found, the process skips to the final aggregation to save compute.

B. Gender-Flipped Variant Generation

Based on detected cues, specialized agents generate alternative translations to create a contrastive evaluation set:

Agent_amb (Ambiguous): For ambiguous sources, generates valid Feminine (F), Masculine (M), and Neutral (N) variants to test if the QE score remains invariant across genders.
Agent_exp (Explicit): For explicit sources, verifies if the original translation aligns with the source gender constraint. If misaligned, it generates a corrected variant; if aligned, it generates a flipped variant to test for preference inversion.

C. Dual-Stream Quality Estimation

FairQE employs a parallel dual-stream approach:

Quantitative Scoring (Agent_qe): A conventional regression-based QE model (e.g., COMETKiwi, MetricX) scores the original translation and all generated variants. This measures volatility (score divergence among semantically equivalent gender variants).
Bias-Mitigating Reasoning (Agent_uqe): An LLM performs explicit reasoning using MQM-style error analysis. It evaluates the original translation against the variants and alignment signals to derive a debiased score ( $q_{uqe}$ ).
- Ambiguous Logic: Checks for score consistency across F/M/N variants.
- Explicit Logic: Validates fidelity to source constraints and penalizes violations.

D. Dynamic Bias-Aware Score Aggregation

The final score ( $q_{final}$ ) is a weighted combination of the conventional score ( $q_{orig}$ ) and the LLM reasoning score ( $q_{uqe}$ ).

Bias Quantification: The system calculates a Bias Score ( $B$ ) based on the volatility of the conventional model's scores across variants (Ambiguous Bias) and preference violations in explicit cases (Explicit Bias).
Dynamic Gating: A soft-gating mechanism determines the weight ( $w$ $w$ ) of the LLM score.
- If bias is negligible ( $B \approx 0$ ), the system relies on the precise, segment-level scoring of the conventional QE model.
- If bias is high, the system dynamically increases reliance on the LLM's unbiased reasoning.
- Formula: $q_{final} = w \cdot q_{uqe} + (1 - w) \cdot q_{orig}$ , where $w$ is a function of the calculated bias score.

3. Key Contributions

FairQE Framework: A model-agnostic, multi-agent architecture that jointly addresses both gender-ambiguous and gender-explicit bias scenarios.
Bias-Aware Dynamic Aggregation: A novel mechanism that quantifies gender bias via gender-flipped variants and dynamically adjusts the final score, preserving the strengths of existing QE models while calibrating their biases.
Comprehensive Evaluation: The first framework to demonstrate that gender bias mitigation can be achieved without sacrificing general QE performance, validated across multiple languages and metrics.

4. Experimental Results

The framework was evaluated on EN–* language pairs (ES, FR, IT, AR, DE, HI) using datasets like GATE, MT-GenEval, and mGeNTE, alongside the WMT 2023 Metrics Shared Task benchmarks.

Gender Fairness (Ambiguous):
- Baseline models showed a feminine-to-masculine score ratio significantly below 1 (indicating masculine bias).
- FairQE achieved ratios closest to 1.0 (parity) across most language pairs, effectively neutralizing the bias.
Gender Fairness (Explicit):
- FairQE improved binary accuracy in identifying gender-aligned translations by up to 18.7 percentage points over the baseline COMETKiwi 22.
- It outperformed GEMBA-MQM (an LLM-as-a-judge baseline), which struggled with inherent LLM gender biases.
General QE Performance:
- On the WMT 2023 EN-DE benchmark, FairQE achieved state-of-the-art or competitive performance in system-level and segment-level accuracy (e.g., improving avg-corr from 0.743 to 0.812 over the backbone).
- This confirms that mitigating bias does not require a trade-off in evaluation accuracy.

5. Significance

Paradigm Shift: FairQE moves beyond simple bias detection to active mitigation within the evaluation pipeline.
Practicality: It is "plug-and-play" and model-agnostic, meaning it can be applied to any existing QE backbone (regression-based or LLM-based) without retraining the core model.
Ethical Impact: By ensuring that translation quality scores are not influenced by gender stereotypes, FairQE enables more equitable model selection and deployment, preventing the reinforcement of societal biases in automated translation systems.
Cost Efficiency: The framework uses a dynamic invocation strategy, reducing API costs significantly (approx. $0.00095 per sample) compared to running all agents on every instance.

In conclusion, FairQE demonstrates that fairness and accuracy are not mutually exclusive in translation quality estimation, offering a robust solution for the next generation of MT evaluation systems.