Technical Summary: MultiSynt/MT

Problem Statement

Open web-scale pre-training corpora remain heavily concentrated in English, creating a significant bottleneck for the development of multilingual Large Language Models (LLMs). While high-quality native data exists for English, most other languages suffer from a lack of volume or quality in openly available resources. Although machine translation (MT) has been proposed as a scalable solution to generate multilingual pre-training data, it raises legitimate concerns regarding "translationese" (stylistic artifacts), the inheritance of the source language's cultural reference frame (e.g., English-anchored names, places, and idioms), and the potential for translated text to underperform native data on culturally nuanced tasks.

Methodology

The authors introduce MultiSynt/MT, an open synthetic parallel corpus designed to address the scarcity of high-quality pre-training data for 36 languages, with a focus on medium- and lower-resource European languages.

Data Construction

• Source Data: The corpus is derived from a uniform random sample of approximately 155 million documents (roughly 100 billion tokens) from Nemotron-CC, a high-quality, non-synthetic Common Crawl split filtered by an ensemble quality classifier. This source was selected because recent reference baselines indicate it produces superior downstream performance compared to established English corpora like C4 or The Pile.
• Translation Systems: To ensure coverage across 36 languages, the authors employed a hybrid pool of open translation systems:
  • TOWER+ Models: A translation-specialized LLM family (9B and 72B parameters) used for 16 and 5 languages respectively.
  • OPUS-MT / HPLT-MT: Classical Neural Machine Translation (NMT) systems used for 31 languages, including those where LLM-based translation is less feasible or available.
• Pipeline: Two parallel pipelines were executed on the same source documents to preserve row-aligned parallelism. The LLM pipeline utilized vLLM on NVIDIA A100 GPUs (Leonardo HPC), while the NMT pipeline used Marian-NMT on AMD MI250x GPUs (LUMI HPC). The resulting corpus contains approximately 4.8 trillion target-language tokens (or ~7.1 trillion tokens if counting each system's output separately).
• Licensing: The corpus is released under a Creative Commons Zero (CC0) license, with row-aligned translations from multiple systems to enable controlled research.

Evaluation Framework

The authors trained reference LLMs (1.7B parameters, Llama-like architecture) on MultiSynt/MT variants and compared them against HPLT 2.0, a strong native multilingual baseline.

• Standard Benchmarks: A broad suite of multilingual benchmarks (e.g., Belebele, MMLU, XNLI, Hellaswag) was used to measure general performance.
• Fluency-Sensitive Evaluation: To detect nuances missed by multiple-choice scoring, the authors employed an LLM-as-a-judge protocol to evaluate the fluency of free-form generations.
• Native-Authored Tasks: A case study on Norwegian tasks (NorIdiom and NorCommonsenseQA) from NorEval was conducted to assess performance on idiomatic and culturally grounded knowledge.
• Embedding Analysis: The authors analyzed the embedding neighborhoods of benchmark items to determine alignment with the training corpora.

Key Results

1. Efficiency and Performance on Standard Benchmarks

• Token Efficiency: Reference LLMs trained on MultiSynt/MT reached the final benchmark scores of the native-data baseline (HPLT 2.0) using approximately 72% fewer pre-training tokens.
• Matched Budget Performance: At a matched 100B-token training budget, MultiSynt/MT models outperformed the native baseline by approximately 15% relative.
• System Independence: The performance gain over native data was largely independent of the specific translation system used (TOWER+ 9B, TOWER+ 72B, or OPUS-MT), as the curves for these variants clustered tightly together on standard benchmarks.

2. Evaluation Blind Spots

• Fluency Sensitivity: While standard multiple-choice benchmarks could not distinguish between the different translation systems, the LLM-as-a-judge evaluation successfully recovered the quality ranking established by human raters (TOWER+ 72B ≥ TOWER+ 9B > OPUS-MT). This suggests standard benchmarks are blind to fluency differences that the trained models inherit.
• Benchmark Alignment: An embedding-space diagnostic revealed that standard benchmark items tend to have more MultiSynt/MT documents among their nearest neighbors than HPLT 2.0 documents. This suggests that translated data provides strong coverage of benchmark-adjacent regions, potentially inflating scores on translation-derived benchmarks.

3. Limitations on Cultural and Idiomatic Tasks

• Norwegian Case Study: On tasks requiring idiomatic knowledge (NorIdiom), models trained on native data consistently outperformed those trained on translated data throughout the training process.
• Reasoning vs. Culture: On NorCommonsenseQA (commonsense reasoning), translated data eventually closed the gap and matched or exceeded native data, suggesting that while translated text lacks local cultural anchoring, it preserves structured reasoning patterns from the high-quality English source.

Significance and Claims

The paper positions MultiSynt/MT as the largest openly available pre-training corpus to date for many European low- and medium-resource languages, exceeding the largest comparable native resource (HPLT 3.0) by more than an order of magnitude in the lowest-resource cases.

The authors claim the following significance:

1. Complementarity, Not Replacement: Translated data should be treated as a complement to native data, not a replacement. It is highly effective for closing the data-scale gap in medium- and lower-resource languages but falls short on tasks requiring deep cultural or idiomatic grounding.
2. Source Quality vs. Translation: The observed performance gains may be driven primarily by the high quality of the Nemotron-CC source rather than the translation operation itself. The paper notes that the comparison varies both the source corpus and the translation step, making it difficult to isolate the specific contribution of translation.
3. Evaluation Reform: The study highlights a critical blind spot in standard multilingual evaluation. Multiple-choice benchmarks fail to detect fluency deficits and cultural misalignments that are recoverable via native-authored benchmarks and fluency-sensitive LLM-as-a-judge protocols.
4. Resource Availability: By releasing row-aligned translations from multiple systems, the corpus enables controlled research on translation system choice, scaling, and the attribution of performance gains to source quality versus translation artifacts.

The authors conclude that while MultiSynt/MT offers a powerful tool for scaling multilingual LLMs, researchers must remain cautious regarding the "English-centric" biases inherited from the source and the limitations of translated data in capturing native cultural nuance.