Technical Summary: Cross-Direction Contamination in Machine Translation Evaluation

Problem Statement

Large Language Models (LLMs) are susceptible to data contamination, where public benchmark test sets are inadvertently included in training data. This leads to inflated evaluation scores that reflect memorization rather than genuine generalization. In multilingual settings, this issue is exacerbated by cross-directional contamination: a phenomenon where memorization of a target language in one translation direction artificially boosts performance in unseen translation directions sharing the same target. Current evaluation practices often fail to detect this asymmetry, assuming that contamination requires exact source-input overlap.

Methodology

The authors utilize the FLORES-200 benchmark, a multi-way parallel test suite covering hundreds of languages, as a diagnostic tool. The study employs two primary models:

1. Bloomz-7B1: A multilingual LLM explicitly documented as being fine-tuned on FLORES data (the contaminated model).
2. Llama-3.1-8B-Instruct: A comparable instruction-tuned model with no reported use of FLORES in its training (the uncontaminated control).

The experimental design involves three main phases:

1. Baseline Contamination Detection: Evaluating both models across all 15 selected language pairs (spanning high, medium, and low-resource languages) using BLEU (surface overlap) and COMET (semantic similarity). High BLEU coupled with high COMET is used as a proxy for memorization.
2. Source Perturbation Analysis: To determine if recall relies on exact source strings, the authors introduce:
   • Back-translation: Generating source inputs via a third language using Llama.
   • Paraphrasing: Rewriting source inputs using Llama.
   • Named Entity Replacement: Systematically replacing named entities (e.g., persons, locations, dates) in the source text with new, valid entities of the same type, translated into the target language.
3. Controlled Fine-Tuning: To isolate the mechanism of contamination, the authors fine-tune the clean Llama model on FLORES English-to-X pairs and test its performance on unseen directions (Z-to-Y) to simulate and verify cross-directional leakage.

Key Findings

1. Confirmation of Cross-Directional Contamination

The study confirms that Bloomz exhibits severe contamination on FLORES. Crucially, this contamination is asymmetric:

• Target-Side Memorization: Inflated scores occur primarily when a language is on the target side, regardless of the source language. For instance, Bloomz achieves high BLEU scores (40–90) for various sources translating into English, French, or Tamil, even if the specific source-target pair was not explicitly seen during training.
• Unseen Directions: The model performs moderately well in unseen translation directions (e.g., low-resource source to high-resource target) solely due to the memorization of the target-side text.
• Control Validation: Llama shows no such patterns, with scores dropping significantly when source-target pairs are not high-resource or when the target language is low-resource (e.g., Bambara, Ewe).

2. Recall Triggers Beyond Exact Input

Contrary to the assumption that contamination requires exact source-text matching, the study finds that recall persists even with altered inputs:

• Back-translation and Paraphrasing: When source inputs were back-translated or paraphrased (resulting in low surface similarity to the original source), Bloomz still retrieved memorized target references with moderate to high BLEU scores (20–60). This suggests that specific semantic features or structural cues in the source act as "triggers" for memorized targets, rather than the exact string itself.
• Asymmetry of Triggers: While back-translated sources from low-resource languages (e.g., Bambara) failed to trigger recall, sources from medium-resource languages often did, indicating that the "trigger" mechanism is sensitive to the quality of the back-translation.

3. Named Entity Replacement as a Diagnostic

The most robust finding is the impact of Named Entity Replacement:

• Replacing named entities in the source text consistently causes a significant drop in BLEU scores (5–10 points for single entities, 10–20 points for all entities) for the contaminated model (Bloomz).
• This drop occurs even when the paraphrased or back-translated sources did not reduce recall.
• The authors conclude that named entities serve as critical anchors for memorization. Their removal disrupts the model's ability to retrieve the specific memorized reference, making entity replacement an effective probing method for detecting contamination.

4. Simulation via Fine-Tuning

Fine-tuning the clean Llama model on FLORES English-to-X pairs successfully replicated the Bloomz behavior. The fine-tuned model showed artificial boosts in unseen directions (Z-to-Y), confirming that the cross-directional effect is driven by target-side memorization during the training process, not by pre-existing model architecture quirks.

Significance and Claims

The paper claims to be the first systematic examination of cross-lingual and cross-directional contamination in machine translation. Its primary contributions are:

1. Demonstrating Asymmetry: Proving that contamination is not a symmetric property of a language pair but is driven by target-side memorization, which can artificially inflate scores for any source language translating into a memorized target.
2. Challenging Input Similarity Assumptions: Showing that contamination can persist even when the source input is significantly altered (via back-translation or paraphrasing), provided certain triggers (like named entities) remain intact.
3. Proposing a Diagnostic Tool: Introducing named entity replacement as a practical, effective method for practitioners to probe and verify memorization in potentially contaminated models on multi-way parallel benchmarks.

The authors recommend that practitioners verify contamination across all translation directions, particularly checking for target-side memorization, rather than relying solely on standard evaluation metrics which may mask the extent of data leakage. The study concludes that without such rigorous probing, benchmark scores in multilingual LLMs may reflect data leakage rather than true generalization capabilities.