Technical Summary: CzechDocs

Problem Statement

Despite advancements in neural machine translation (NMT), preserving document structure and markup tags during translation remains a significant hurdle for practical deployment. Real-world translation workflows often involve complex structured documents (HTML, XML, DOCX, PDF) where maintaining the integrity of formatting is essential for downstream processing and publication. This challenge is particularly acute in scenarios requiring rapid multilingual deployment, such as humanitarian crisis communication. The 2022 influx of Ukrainian refugees into the Czech Republic highlighted an urgent need for high-quality translation of critical information (legal documents, government forms, educational materials) while strictly preserving the markup structure necessary for web accessibility and publication. Existing evaluation metrics (e.g., BLEU, COMET) focus primarily on lexical and semantic accuracy, often neglecting the structural fidelity required for production-grade translation systems.

Methodology

Dataset Construction (CzechDocs)

The authors present CzechDocs, a curated multiway parallel dataset designed specifically for evaluating format-preserving machine translation.

• Sources: Data was collected from 14 public Czech-language websites, including educational portals, expat-focused information sites, government services, and municipal portals.
• Content: The corpus focuses on civic integration materials for foreigners, primarily Ukrainian immigrants, with additional content in Vietnamese, English, Russian, and other minority languages.
• Formats: The dataset includes 77 unique documents in three formats:
  • HTML (79%): Manually edited and aligned at the segment level to ensure precise correspondence between source and target XLIFF segments.
  • PDF (17%) and DOCX (4%): Aligned at the document level (verified as translations of each other), though internal segment boundaries may differ.
• Preprocessing: A pipeline was employed involving cleanup (ensuring exact translations with matching formatting), format conversion (PDF to DOCX), and extraction using the Okapi Tikal toolkit to generate XLIFF, Moses InlineText (MOS), and plaintext formats.
• Alignment: For HTML documents, a custom tool was used to enforce segment-level alignment across all language versions, ensuring consistent segment counts and boundaries by using the Czech source as a skeleton.
• Statistics: The final corpus contains 60,153 translatable segments across 15 languages, totaling 271,111 words and 126,833 markup tags. Notably, 90.3% of segments contain at least one markup tag.
• Splits: The dataset is split into a validation set (19 HTML documents, publicly released) and a held-out test set (reserved for a future shared task).

Experimental Evaluation

The authors conducted a comparative evaluation of different strategies for handling translation with markup, utilizing two Large Language Models (LLMs): Aya-expanse-8b (CohereForAI) and gpt-4.1-nano (OpenAI). The study compared the following approaches:

1. Detag-and-Project: Translating plain text only, then reinserting tags into the target using word alignments (via awesome-align and heuristics in Charles Translator).
2. Direct Tagged Input: Feeding full, tagged segments (in Moses InlineText format) directly to the model with a standard translation prompt.
3. Prompt with Tag Emphasis: Similar to (2), but with explicit instructions in the prompt to preserve tag positions accurately. This included variations with and without context (previous document segments).

Performance was measured using BLEU scores for both tagged text (relative to tagged references) and untagged plain text (relative to untagged references).

Key Contributions

1. A Curated Dataset: The release of CzechDocs, a specialized parallel corpus of formatted documents (HTML, DOCX, PDF) covering Czech and minority languages used in Czechia. The dataset is explicitly designed for evaluating tag-aware translation and localization.
2. Comparative Analysis: A systematic evaluation of common strategies for format-preserving translation, including LLM-based direct translation with various prompting strategies versus traditional detag-and-project heuristics.

Results

The evaluation on the validation set yielded the following observations:

• Model Performance: The gpt-4.1-nano model generally demonstrated slightly better performance than Aya-expanse-8b.
• Prompt Sensitivity: The models were sensitive to prompt engineering. Explicit instructions to preserve markup led to noticeable improvements in tag preservation scores, particularly for gpt-4.1-nano.
• Approach Comparison:
  • Tagged BLEU: The "Detag-and-Project" approach and the "Tag-aware Prompting" approaches achieved comparable results on average. However, the Detag-and-Project method showed greater variance across individual documents.
  • Untagged BLEU: The Detag-and-Project approach resulted in slightly lower untagged BLEU scores compared to direct LLM translation, suggesting that the alignment-based reinsertion process may introduce minor semantic deviations or that the LLMs handle the raw text slightly better when not constrained by tag reconstruction logic.
• Specific Findings: For gpt-4.1-nano, the "Explicit" prompt (MOS input) achieved a Tagged BLEU of 87.8, significantly outperforming the "Normal" prompt (25.4). The "Detag-and-Project" approach with gpt-4.1-nano achieved a Tagged BLEU of 89.0.

Significance and Claims

The paper positions CzechDocs as a necessary resource for advancing the evaluation of machine translation systems that must operate in real-world, structured environments. The authors claim that while LLMs are capable of translating tagged segments directly without specialized fine-tuning, the traditional "detag-and-project" approach remains competitive, particularly in terms of automated metrics on tagged text.

The authors modestly state that their work demonstrates a "possible use-case" for the dataset and provides a baseline comparison of existing strategies. They do not claim that LLMs have solved the problem of format preservation; rather, they highlight that explicit prompting improves tag preservation and that the choice between direct LLM translation and alignment-based reinsertion involves trade-offs between tag fidelity and plain-text fluency. The dataset is intended to support future research and a shared task focused on document-level translation with formatting preservation.