HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

1. Problem Statement

The development of Large Language Models (LLMs) and Machine Translation (MT) systems is currently dominated by massive, high-quality pre-training datasets. However, these resources are largely:

• English-centric: Most major datasets (e.g., C4, FineWeb) heavily favor English, leaving hundreds of languages underrepresented.
• Proprietary or Corporate: High-quality "refined" datasets are often produced by corporate entities (Google, NVIDIA, Meta), limiting transparency and accessibility for the academic community and smaller organizations.
• Lacking in Diversity: Existing multilingual datasets often lack deep linguistic diversity, covering only major language families or scripts.
• Insufficiently Documented: The processes for cleaning, filtering, and annotating web-scale data are often opaque, making it difficult to replicate results or understand data biases.

The HPLT consortium aims to "democratize" the LLM landscape by providing an open, transparent, and massive multilingual resource that rivals corporate offerings in scale and quality.

2. Methodology

The paper outlines a comprehensive, open-source pipeline for processing raw web archives into high-quality training data. The methodology involves several key stages:

A. Data Sourcing

• Sources: The dataset aggregates 7.2 petabytes of raw web data, combining:
  • Internet Archive (IA): "Wide crawls" from 2012–2020.
  • Common Crawl (CC): 57 full snapshots from 2014–2025 (including all data since 2020).
  • ArchiveBot: A minor sample (planned expansion in HPLT 4.0).
• Scale: This represents a five-fold increase in raw CC data compared to the previous HPLT 2.0 release.

B. The Processing Pipeline (Monotextor & Bixtextor)

The pipeline is an evolution of the HPLT 2.0 system, featuring:

1. Text Extraction: Uses Trafilatura with optimized hyperparameters prioritizing extraction quality over speed.
2. Language Identification (LID): Uses OpenLID-v2, updated to align with the Flores+ evaluation set and handle noisy web orthography via normalization (lowercasing, whitespace handling).
3. Deduplication:
   • Implements MinHash-based global near-deduplication for all languages except English, Russian, and Chinese (which use per-crawl deduplication for efficiency).
   • This significantly increases text diversity compared to per-crawl methods.
4. Annotation & Filtering:
   • Web Docs Scorer (WDS): A heuristic tool scoring documents based on length, "oddity" signals, and language distribution.
   • Register Classification: A retrained classifier assigns register labels (e.g., News, Blog, Encyclopedia) to 104 languages.
   • Safety & PII: Filters for adult content, artifacts, and Personally Identifiable Information (PII) using tools like TruffleHog and multilingual PII detectors.
5. Packaging: Data is distributed as Zstandard-compressed JSONlines, organized by WDS quality bins to allow flexible sampling.

C. Evaluation Framework (HPLT-e)

To validate data quality, the authors developed HPLT-e, a framework for automated, large-scale multilingual evaluation:

• Benchmarks: Covers 9 European languages with 127 tasks (entailment, reasoning, toxicity, etc.).
• Prompt Robustness: Uses 500+ diverse prompts per task to mitigate prompt sensitivity.
• Aggregation: Uses min-max normalization and Borda's count to aggregate scores across heterogeneous tasks and languages.

D. Model Training

• Monolingual Models: Trained ~60 GPT-BERT (encoder) and T5-like (encoder-decoder) models across nearly 60 languages.
• Decoder Models: Pretrained 2.15B parameter models on 100B tokens from various datasets (FineWeb, HPLT 2.0/3.0, MADLAD-400) to compare performance.
• Synthetic Data: Generated parallel data by translating high-quality English documents (from FineWeb-Edu and Nemotron-CC) into 36 target languages using OPUS-MT models.

3. Key Contributions

1. HPLT 3.0 Dataset:

   • Scale: 30 trillion subword tokens (using the Gemma-3 tokenizer), making it likely the largest generally available multilingual pre-training dataset.
   • Coverage: Covers ~200 language-script combinations, with nearly 50% of tokens being non-English.
   • Quality: Features rich metadata, including WDS quality scores, register labels, and PII flags.

2. Open Infrastructure:

   • Full release of the open-source processing pipeline (Monotextor/Bixtextor).
   • Public availability of ~60 monolingual encoder/encoder-decoder models and reference decoder models.
   • Bilingual Resources: ~1.1 billion sentence pairs and a document-level parallel corpus spanning 406 language pairs.

3. Synthetic Data Generation:

   • Creation of synthetic parallel corpora (approx. 3.3 trillion tokens total) by translating high-quality English data into under-resourced languages, providing a novel method for data augmentation.

4. Evaluation Framework:

   • Introduction of HPLT-e, a rigorous benchmarking suite designed to test data quality via model performance, emphasizing prompt robustness and native speaker validation.

4. Results

Data Quality & Analytics

• Diversity: Global deduplication increased the ratio of unique segments from 52% (HPLT 2.0) to 73% (HPLT 3.0).
• Domain Shift: Reduced over-representation of Wikipedia (from >25% in many datasets to only 7 datasets in HPLT 3.0) and increased variety in top-level domains (TLDs).
• Manual Inspection: For 23 languages, manual checks showed pornographic content <2% for most languages. Language identification errors were low, with notable exceptions for closely related languages (e.g., Bosnian/Serbian).

Model Performance (LLM Pre-training)

• Comparison: Models pretrained on HPLT 3.0 outperformed those trained on FineWeb and HPLT 2.0 on the HPLT-e benchmarks.
• Ranking: HPLT 3.0 models ranked 2nd overall, behind only MADLAD-400 (which is older and smaller), demonstrating that HPLT 3.0's refined quality yields competitive performance gains.
• Quality Sampling: Experiments showed that training on high WDS-scored data improves performance, while training on low WDS data degrades it.

Monolingual Model Performance

• GPT-BERT & T5: The HPLT 3.0 trained models consistently outperformed baselines like mBERT, XLM-R, and mmBERT on Universal Dependencies tasks (POS, Lemmatization, Parsing) and Named Entity Recognition (NER).
• Linguistic Competence: On the MultiBLIMP benchmark (linguistic minimal pairs), HPLT 3.0 GPT-BERT models achieved state-of-the-art results, often surpassing the best systems reported in previous literature (e.g., Goldfish).
• Encoder-Decoder: The T5-like models outperformed the multilingual mT5-base on both NER and linguistic competence tasks, proving the value of language-specific training on recent, high-quality data.

Machine Translation

• Baseline NMT models trained on the mined parallel data achieved strong BLEU and COMET scores on the FLORES200 benchmark, validating the utility of the extracted parallel data for MT tasks.

5. Significance

• Democratization: HPLT 3.0 challenges the monopoly of corporate entities on large-scale language data, providing a transparent, publicly funded alternative under permissive licensing (CC0 for metadata, dual licensing for text).
• Multilingual Equity: By covering ~200 languages with high-quality data, it significantly advances NLP capabilities for low-resource and medium-resource languages, addressing the "digital divide."
• Scientific Reproducibility: The release of the full pipeline, intermediate checkpoints, and detailed analytics allows the community to replicate, audit, and improve upon the data curation process.
• Future Research: The dataset serves as a foundation for studying the effects of data quality, register diversity, and synthetic data on LLM performance, while the upcoming HPLT 4.0 will release the raw, unfiltered data pool for further research.

In conclusion, HPLT 3.0 represents a landmark achievement in open-source NLP, proving that academic consortia can produce data resources that rival or exceed corporate offerings in scale and quality, specifically tailored to support a truly multilingual AI future.