Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study

Technical Summary: Data Contamination in Neural Hieroglyphic Translation

Problem Statement
Ancient and endangered languages, such as Ancient Egyptian, present unique challenges for Natural Language Processing (NLP) due to extreme data scarcity. Unlike modern languages, datasets for these domains rely on surviving archaeological artifacts and require specialist annotation, making them difficult to expand. This scarcity renders data quality issues disproportionately consequential. The paper investigates a specific case of hieroglyphic-to-German translation where a recent study (De Cao et al., 2024) reported a state-of-the-art score of 61.5 BLEU using a fine-tuned M2M-100 model. The authors found a significant discrepancy when reproducing these results with the publicly released model, achieving only 37.0 BLEU. This gap suggested a potential methodological flaw, specifically data contamination, where evaluation targets inadvertently appear in the training data. Ancient corpora are particularly susceptible to this due to the pervasive use of formulaic repetition (e.g., medical instructions, offering formulae, royal epithets) across texts, which standard random splitting fails to address.

Methodology
The study conducted a comprehensive reproducibility and contamination audit using the hiero-transformer repository data.

• Data Source: The authors utilized the Thesaurus Linguae Aegyptiae (TLA) dataset. After filtering for entries with both digitized hieroglyphic sources (in Gardiner notation) and non-empty German targets, the usable training set contained 18,669 pairs, and the test set contained only 50 valid samples.
• Contamination Detection: The authors employed two primary detection methods:
  1. Normalized Exact Match: Test targets were compared against all unique training targets after normalization (Unicode NFC, lowercasing, punctuation removal, whitespace collapse). This revealed 16 out of 50 test samples (32%) had targets identical to training data.
  2. Character n-gram Overlap: To detect "soft" leakage, the authors calculated the fraction of character 8-grams in test targets appearing in training targets. At a 70% overlap threshold, 50% of test samples were flagged.
• Experimental Setup: Five model configurations were evaluated across two architectures (M2M-100 and mBART-50):
  1. The original released model (De Cao et al., 2024).
  2. A script reproduction of the original training procedure.
  3. Three models trained with a modernized, low-resource recipe (AdamW, cosine schedule, label smoothing, gradient clipping, and data upsampling).
• Evaluation Protocol: Performance was measured using BLEU (case-insensitive) and chrF++ on three subsets: the full test set (50 samples), the contaminated subset (16 samples), and the clean subset (34 samples). COMET-22 was also used for validation.

Key Results
The investigation confirmed that data contamination is the primary driver of inflated performance metrics in this domain:

• Performance Gap: Across all five models, contaminated samples scored 29 to 47 BLEU points higher than clean samples. For instance, the best-performing model (M2M-100 Hybrid) achieved 83.8 BLEU on contaminated samples versus 37.3 BLEU on clean samples.
• Metric Consistency: The inflation was consistent across architectures and metrics. Contaminated samples scored 0.23–0.26 COMET-22 points higher than clean samples.
• Memorization vs. Translation: A retrieval baseline that simply copied the best-matching training target achieved 81.8 BLEU on exact-match items, rivaling the best neural model (77.9 BLEU), while scoring 0.0 on clean items. This indicates that high scores on contaminated data stem from target memorization rather than genuine translation capability.
• Source Independence: Analysis showed that target memorization occurs independently of source input. One test item with zero source overlap to any training sample still achieved 100% target overlap, and 10 of the 16 exact-match items had training sources with less than 5% similarity.
• Decontamination Limits: A document-level decontamination experiment (removing all training sentences from the 33 source documents of the test set) reduced the BLEU score on exact-match items by only 4.6 points. This is because 8 of the 16 contaminated targets persisted in the training data via other source documents (e.g., the phrase "Werde gekocht" appearing across multiple independent papyri). This demonstrates that document-level splitting is insufficient; target-level deduplication is required.

Key Contributions

1. First Contamination Audit for Ancient-Language NMT: The study identifies that formulaic repetition in ancient corpora creates systematic data-leakage risks absent in modern benchmarks, revealing a 32% exact target overlap in a public test set.
2. Cross-Architecture Robustness: The authors demonstrate that the performance inflation is dataset-inherent, not model-specific, as the gap persists across M2M-100 and mBART-50 architectures and various training regimes.
3. Graduated Decontamination Analysis: The paper characterizes contamination severity through exact matching, character 8-gram overlap, document-level leakage analysis, and per-item frequency cataloging.
4. Corrected Baselines and Protocol: The authors release a decontaminated test set of 34 samples (with no exact target overlap) and establish corrected baselines of 30.9–39.2 BLEU. They also propose a splitting protocol involving document-level partitioning, target deduplication, and genre stratification.

Significance and Claims
The paper claims that the reported 61.5 BLEU score for hieroglyphic-to-German translation is an artifact of data contamination rather than a reflection of true model capability. The corrected baselines (30.9–39.2 BLEU) provide a realistic assessment of Neural Machine Translation (NMT) for this endangered writing system. The authors argue that while NMT can capture the general gist of ancient texts (useful for corpus triage), it contains significant errors in domain terms and entity identification, making it unsuitable for philological analysis without expert verification.

The study highlights that formulaic repetition is a universal property of ancient corpora (including Latin, Greek, and Akkadian), suggesting that contamination is a likely, systematic issue in ancient-language NLP benchmarks that has been previously unaudited. The authors conclude that future benchmarks must report contamination statistics and partition results by contamination status to avoid misleading evaluations. They frame their work as a constructive contribution to reproducibility, noting that the identified contamination likely arose from standard preprocessing procedures rather than intentional manipulation.