💬 NLP

Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptation

This paper proposes a two-stage fine-tuning strategy for Large Language Models in document-level machine translation that leverages LLM-augmented and rigorously filtered synthetic corpora to overcome data scarcity and mitigate generation hallucinations.

Original authors: Ireh Kim, Tesia Sker, Chanwoo Kim

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ireh Kim, Tesia Sker, Chanwoo Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced translator (an AI) how to translate entire books, not just single sentences. This is the challenge the paper tackles.

Here is the story of how the researchers fixed the translator's biggest problems using a clever two-step training plan and a strict quality control system.

The Problem: The "Hallucinating" Translator

Think of Large Language Models (LLMs) like a very smart student who has read the whole library but has never practiced translating a whole story.

The Good: They are great at understanding context. If you say, "He picked up the apple," and later say, "He ate the fruit," the student knows they are the same thing. This is crucial for translating documents where sentences flow into each other.
The Bad: When asked to translate a whole document, this student tends to daydream. They might invent details that weren't in the original text (hallucinations) or forget important parts entirely (omissions).
The Missing Piece: We don't have enough "textbooks" (high-quality, human-translated documents) to teach them properly. Most existing data is just isolated sentences, like flashcards, which doesn't teach them how to handle a whole story.

The Solution: A Three-Part Recipe

The researchers came up with a plan to fix this using Synthetic Data (AI-made data), Quality Filtering, and Two-Stage Training.

Step 1: Making Up the Textbooks (Data Augmentation)

Since they didn't have enough real textbooks, they decided to write their own.

The Analogy: Imagine they took a pile of news articles (the CNN/Daily Mail dataset) and asked a super-smart AI (Llama 3.1) to translate them into German.
The Risk: Because the AI is prone to daydreaming, these new translations might be full of made-up facts or missing details. It's like a student writing a translation from memory without checking the source.

Step 2: The Strict Quality Control (Multi-Metric Filtering)

You can't just use the AI's homework; you have to grade it first. The researchers set up a "Quality Control Gate" with three different inspectors:

The Word Counter (sacreBLEU): Checks if the words match the source closely.
The Human Judge (COMET): A smart metric that tries to guess how a human would rate the translation.
The Meaning Matcher (LaBSE-CosSim): This is the most important one. It checks if the meaning is the same, even if the words are different. It's like checking if the "vibe" of the translation matches the original.

The Process: They generated thousands of translations, ran them through these three inspectors, and threw away anything that didn't pass the test. This left them with a "Gold Standard" dataset of clean, high-quality document translations.

Step 3: The Two-Stage Training Camp

Now that they had good data, they trained the model in two phases, like a martial arts student.

Stage 1: The Basics (Sentence-Level Training)
- Analogy: Before learning to run a marathon, you learn to walk. They first trained the AI on millions of simple, single-sentence translations (using standard datasets like News Commentary). This taught the model the basic grammar and vocabulary without the pressure of a long story.
Stage 2: The Advanced Class (Document-Level Training)
- Analogy: Now that the student can walk, it's time to run the marathon. They took the model from Stage 1 and fine-tuned it on the filtered document-level data they created in Step 2.
- Why this works: Because the model already knew the basics, it could now focus entirely on learning how to keep the story consistent across paragraphs without getting confused or making things up.

The Results: A Smarter Translator

The experiments showed that this method worked like a charm:

Better than guessing: The two-stage approach was much better than just throwing the model into the deep end (training only on document data).
Quality matters: The more strictly they filtered the data (using all three inspectors), the better the final translator performed.
The Sweet Spot: The best results came from a specific combination of filters: high word overlap, high human-quality scores, and high semantic similarity.

The Takeaway

This paper proves that you don't need millions of expensive, human-translated books to teach an AI how to translate documents. Instead, you can:

Generate your own practice material.
Filter it ruthlessly to remove errors.
Train the AI in stages (basics first, then advanced).

It's like taking a raw diamond, cutting away the rough edges with a precise filter, and then polishing it with a two-step process to reveal a gem that shines brighter than before.

1. Problem Statement

While Large Language Models (LLMs) excel at modeling contextual information, making them theoretically ideal for Document-Level Machine Translation (MT), their practical adoption has been limited compared to conventional encoder-decoder systems. The paper identifies two primary bottlenecks:

Data Scarcity: There is a lack of large-scale, high-quality parallel document-level data. Existing datasets (e.g., News Commentary, Europarl) are often sentence-aligned or too small for effective LLM fine-tuning.
Generation Errors: LLMs are prone to hallucinations (adding information not present in the source) and omissions (dropping source information) during translation. These errors are not always captured by traditional metrics like sacreBLEU, leading to unreliable translations.

2. Methodology

The authors propose a comprehensive framework consisting of Data Augmentation, Multi-Metric Filtering, and a Two-Stage Fine-Tuning Strategy.

A. Data Augmentation Pipeline

To overcome data scarcity, the authors generate synthetic document-level parallel data:

Source: They utilize the CNN/Daily Mail summarization dataset, chosen for its coherent narrative structure and journalistic style, which provides sufficient textual context.
Generation: A large LLM (Llama-3.1-8B-Instruct) translates English documents into German.
Pseudo-Reference Creation: To evaluate the quality of the LLM output, they generate "pseudo-references" using Google Translate (a conventional sentence-level MT system known for fewer hallucinations).

B. Multi-Metric Filtering

To mitigate the hallucinations and omissions introduced during synthetic generation, the augmented data is filtered using three distinct metrics:

sacreBLEU: Measures n-gram overlap with the pseudo-reference.
COMET: A learned, reference-based metric that predicts human quality judgments.
LaBSE-CosSim: A reference-free metric calculating cosine similarity between sentence-level embeddings of the source and translation (averaged for document-level). This is specifically effective for detecting semantic hallucinations.

Note: The authors rejected xCOMET and Doc-COMET because they either lack document-level input support or require sentence-level alignment unavailable in their augmented data.

C. Two-Stage Fine-Tuning Strategy

The authors introduce a specific training curriculum to stabilize the model:

Stage 1 (Sentence-Level): The base model (Llama-3.2-1B-Instruct) is first fine-tuned on abundant, high-quality sentence-level data (News Commentary v16 and Europarl). This establishes a strong foundation for translation accuracy.
Stage 2 (Document-Level): The model is subsequently fine-tuned on the filtered synthetic document-level corpus. This stage adapts the model to utilize long-range context while minimizing the risk of hallucinations due to the prior filtering.

3. Key Contributions

Synthetic Data Construction: A novel pipeline converting summarization datasets into document-level MT pairs using LLMs, addressing the scarcity of En-De document-level data.
Robust Filtering Framework: The integration of LaBSE-CosSim alongside traditional metrics (sacreBLEU, COMET) to specifically target semantic hallucinations and omissions in synthetic data.
Curriculum Learning for MT: Demonstration that a two-stage fine-tuning approach (Sentence $\to$ Document) significantly outperforms direct document-level training, effectively balancing fluency and contextual consistency.

4. Experimental Results

The experiments focused on English-to-German (En $\to$ De) translation using the Llama-3.2-1B-Instruct model.

Impact of Two-Stage Strategy:
- Models trained using the two-stage strategy significantly outperformed those trained only on document-level data.
- Metrics: sacreBLEU improved from 11.24 to 15.07; COMET from 0.618 to 0.697; and LaBSE-CosSim from 0.764 to 0.852.
Impact of Filtering:
- Filtering augmented data consistently improved performance across all metrics.
- Single Metric: sacreBLEU-based filtering showed the most notable individual improvement.
- Combined Metrics: Combining metrics yielded the best results. The synergy between lexical overlap (sacreBLEU) and semantic evaluation (COMET/LaBSE-CosSim) provided complementary filtering effects.
- Optimal Configuration: The best performance was achieved using all three metrics with the following thresholds:
  - sacreBLEU $\ge$ 35
  - COMET $\ge$ 0.75
  - LaBSE-CosSim $\ge$ 0.85
- Final Scores: sacreBLEU 15.96, COMET 0.701, LaBSE-CosSim 0.860.

5. Significance and Conclusion

This work demonstrates that LLMs can be effectively adapted for document-level MT even under resource constraints, provided that:

Data Quality is Enforced: Synthetic data must be rigorously filtered using a combination of reference-based and reference-free metrics to remove hallucinations.
Training is Structured: A two-stage curriculum (starting with sentence-level data) is crucial for stabilizing the model before introducing complex document-level context.

The proposed framework offers a scalable solution for generating high-quality document-level translation data and suggests that these methods can generalize to other language pairs and domains, bridging the gap between the contextual capabilities of LLMs and the reliability required for practical MT applications.