Technical Summary: Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

Problem Statement

Data contamination—where benchmark content appears in a model's training corpus—undermines the validity of Large Language Model (LLM) evaluation by allowing models to rely on memorization rather than genuine generalization. While detection methods exist, they are predominantly limited to English benchmarks and often fail to account for multilingual dynamics.

This paper investigates a specific blind spot: whether translating benchmarks into a low-resource language (Arabic) acts as a natural barrier to contamination or merely obscures memorization effects. The authors hypothesize that translation might suppress conventional contamination indicators while models still disproportionately benefit from exposure to the translated data, particularly if the model possesses strong capabilities in the target language.

Methodology

Experimental Setup

The study fine-tuned four open-weight instruction-tuned models (Llama-3.2-1B, Mistral-7B, Gemma-3-1B, and Qwen3-1.7B) using parameter-efficient fine-tuning (LoRA). The training conditions involved mixing original English training data with varying proportions (0%, 10%, 50%, 100%) of Arabic-translated test sets from two benchmarks:

• MMLU: Multiple-choice questions (MCQ).
• XQuAD: Extractive Question Answering (QA).

Models were evaluated on the original English splits of these benchmarks.

Detection Mechanisms

To disentangle reasoning from memorization, the authors employed three complementary signals:

1. TS-Guessing (Tested Slot Guessing): An extended method involving choice reordering and masking.
   • For MMLU: Answer choices are shuffled, and one option is masked. The model is prompted to predict the masked text or the original index. High "Index Recall Rate" (IDR)—predicting the original letter despite shuffling—indicates reliance on memorized positional patterns.
   • For XQuAD: Critical tokens in questions are masked. Exact Match (EM) or high ROUGE-L F1 scores on the masked token suggest benchmark leakage.
2. Min-K++ Probability Analysis: A likelihood-based probe measuring the concentration of low-perplexity tokens. Unusually confident token probabilities (low Min-K++ scores) relative to held-out data serve as distributional evidence of memorization.
3. Translation-Aware Contamination Detection (TACD): A proposed diagnostic protocol that does not require access to training data. It constructs multiple evaluation views for each instance:
   • Cross-lingual views: Translations into English, Arabic, and French.
   • Structural perturbation: Random permutation of answer choices within each language view.
   • Signals: It computes Index Recall Rate (IDR) and Cross-Lingual Consistency (CLC) (the fraction of instances where predictions are identical across all language views).

Key Results

Performance and Traditional Signals

• MMLU: Accuracy increased monotonically with contamination levels across all models. TS-Guessing revealed elevated IDR for LLaMA and Qwen at high contamination levels, indicating reliance on memorized answer positions. Min-K++ effectively separated clean from contaminated models (AUROC dropped from ~0.6 to ~0.33–0.44).
• XQuAD: Trends were heterogeneous. While Gemma and LLaMA showed gradual improvement, Mistral degraded beyond 10% contamination. TS-Guessing showed low exact match rates but increasing ROUGE-L F1, suggesting partial recovery of masked content. Min-K++ detection was weaker and less consistent for XQuAD compared to MMLU.

The Translation Effect

The study found that translating benchmarks into Arabic suppressed conventional contamination indicators (such as standard English-only checks) yet models still benefited from the exposure.

• TACD Findings:
  • LLaMA: Showed low IDR and gradually increasing cross-lingual consistency with poisoning.
  • Qwen: Exhibited perfect cross-lingual consistency (1.0) across all poisoning levels while maintaining near-chance IDR, suggesting a model-specific collapse where predictions are weakly conditioned on input content.
  • Gemma: Displayed a distinct regime where IDR remained near random baseline, but cross-lingual consistency increased monotonically with poisoning. This indicates that poisoned exposure induced representational invariance across translations without manifesting as explicit index recall—a signal invisible to IDR alone.

Key Contributions

1. Evidence of Obscured Contamination: The paper demonstrates that translating benchmarks into Arabic can mask traditional contamination signals while still inflating performance on original English benchmarks. This reveals a critical blind spot in current English-centric evaluation pipelines.
2. Extension of Detection Methods: The authors extend TS-Guessing with a choice-reordering strategy and integrate Min-K% probability analysis to capture both behavioral (positional) and distributional (likelihood) contamination signals.
3. Proposal of TACD: The paper introduces Translation-Aware Contamination Detection (TACD), a protocol that identifies contamination by comparing signals across multiple translated benchmark variants (English, Arabic, French) and structural perturbations. TACD successfully exposes contamination even when English-only methods fail, particularly by detecting cross-lingual invariance and model-specific behavioral regimes.

Significance and Claims

The authors claim that their findings highlight the necessity for multilingual, translation-aware evaluation pipelines. They argue that translation should not be treated as a decontamination strategy. Instead, contamination auditing must explicitly compare signals across translated variants and pair aggregate metrics with behavioral probes robust to surface-form changes.

The paper concludes that to ensure fair, transparent, and reproducible assessment of LLMs, the community must move beyond English-only assumptions. TACD is presented not as a definitive proof of contamination, but as a practical, reproducible tool to distinguish reasoning-driven consistency from memorization-consistent behavior in multilingual settings.