Learning from Many Voices: Literary MT Using Multi-Reference Human and Synthetic Data

Technical Summary: Learning from Many Voices: Literary MT Using Multi-Reference Human and Synthetic Data

Problem Statement
Literary machine translation (MT) presents unique challenges distinct from non-literary domains like news or academic writing. Literary texts are deeply embedded in cultural and historical contexts, often employing devices such as metaphor, rhythm, and symbolism that allow for multiple valid interpretations. Consequently, a single source text naturally possesses multiple valid translations that vary in style, word choice, and syntactic structure while maintaining semantic fidelity. While prior work has explored leveraging multiple references (often synthetically generated via back-translation or paraphrasing) to improve MT, these studies have largely focused on sentence-level, non-literary data. The paper addresses the gap in understanding how to effectively utilize existing multi-reference datasets of expert literary translations and investigates whether synthetic alternatives can match the quality of human expert translations for fine-tuning MT models.

Methodology
The authors propose a framework for filtering and utilizing multi-reference datasets based on semantic similarity.

1. Data Source: The study utilizes the Par3 dataset, which aligns multiple expert English translations of non-English literary works at the paragraph level. Experiments focus on French-to-English and Russian-to-English language pairs.
2. Semantic Similarity Scoring (SEMSIM): To quantify the variation among reference translations, the authors employ a scoring system called SEMSIM. This metric calculates the average pairwise semantic similarity between all reference translations for a given source text using the PARAGRAM-SP model.
   • Low SEMSIM: Indicates high divergence, potentially signaling noise, misalignment, or overly divergent interpretations.
   • Medium/High SEMSIM: Indicates meaningful variation where translations remain faithful to the source meaning but exhibit lexical and syntactic diversity.
3. Filtering Strategy: Source texts are filtered into three groups based on their SEMSIM scores: Low ([-1, 0.45)), Medium ([0.45, 0.85)), and High ([0.85, 1]).
4. Experimental Design:
   • Hypothesis 1: Fine-tuning models on different SEMSIM ranges to determine which yields the best performance.
   • Hypothesis 2: Comparing filtered datasets (Medium + High) against unfiltered datasets to assess if excluding low-quality (low SEMSIM) data improves results.
   • Hypothesis 3: Augmenting single-reference datasets with synthetic paraphrases generated by Large Language Models (LLMs) and comparing performance against datasets using only human multi-references.
5. Models and Metrics: Three models were fine-tuned: mT5-Large, Gemma3-12B-PT, and OpusMT. Performance was evaluated using automatic metrics (BLEU, COMET, chrF++) and human evaluations involving pairwise A/B comparisons by professional translators.

Key Results

• Impact of Semantic Similarity: Fine-tuning on data with Medium and High SEMSIM scores substantially outperformed Low SEMSIM data across all models and metrics. For instance, moving from Low to Medium SEMSIM resulted in significant BLEU gains (e.g., +77.9% for mT5). The performance gap between Medium and High SEMSIM was negligible for most models, suggesting that both ranges are highly effective.
• Filtering vs. Unfiltered Data: A dataset filtered to include only Medium and High SEMSIM examples achieved performance comparable to or better than the full unfiltered dataset, despite the unfiltered set containing approximately 14% more training instances. This indicates that the inclusion of Low SEMSIM data (which often contains noise or excessive divergence) does not benefit, and may hinder, literary MT training.
• Human vs. Synthetic Data: Synthetic translations generated by LLMs (Gemma3, Mistral, Qwen) did not reliably improve MT performance compared to human expert translations.
  • In automatic metrics, synthetic-augmented datasets generally underperformed or matched single-reference human datasets, failing to reach the performance levels of multi-reference human datasets.
  • In human evaluations, translators significantly preferred outputs from models fine-tuned on human multi-reference data over those trained on synthetic-augmented data (50.4% preference vs. 32.1%).
  • Analysis of semantic similarity scores revealed that synthetic translations often exhibited lower semantic similarity to human references than human references did to each other, indicating a distributional shift away from the quality of expert translations.

Significance and Claims
The paper claims that for literary machine translation, the quality and nature of the reference data are paramount. The authors demonstrate that:

1. Filtering is Essential: Not all multi-reference data is beneficial. A filtering framework based on semantic similarity effectively identifies source texts where references display meaningful variation without sacrificing faithfulness, leading to superior model performance.
2. Human Expertise is Indispensable: While synthetic data is economical, it cannot currently replace the value of human expert translations for fine-tuning literary MT models. Synthetic paraphrases often fail to capture the nuanced stylistic and semantic fidelity required for high-quality literary translation, and in some cases, degrade model performance.
3. Practical Application: The proposed filtering approach allows practitioners to leverage existing multi-reference datasets more effectively, achieving state-of-the-art results with fewer, higher-quality training instances.

The study concludes that while synthetic data generation is a promising avenue for low-resource settings, the current state of LLMs does not yet allow them to replicate the reliability of human expert multi-reference data for the specific domain of literary translation.