GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training

1. Problem Statement

The development of Large Language Models (LLMs) for low-resource languages, specifically Dutch, faces significant hurdles:

• Data Scarcity & Quality: While web-crawled corpora (e.g., Common Crawl) offer scale, they often suffer from poor quality, noise, and unclear licensing.
• Legal Uncertainty: Copyright legislation (particularly in the EU) and the "grey area" of using scraped data for LLM training create legal risks. Many existing datasets contain Non-Commercial (NC) or Share-Alike (SA) licenses, which restrict commercial use or derivative works, making them unsuitable for commercial AI development.
• Bias and Harm: Unfiltered web data often contains discriminatory, offensive, or outdated content, leading to biased model outputs.
• Dependency: Dutch-speaking regions currently rely on multilingual models where Dutch is not the primary focus, resulting in suboptimal performance.

2. Methodology

The authors developed the GPT-NL Public Corpus by adhering to a strict three-pronged framework: Usefulness, Lawfulness, and Harm Reduction.

A. Data Selection Guidelines

1. Usefulness: The corpus prioritizes Dutch and English. It also includes German, Danish, and Frisian to leverage syntactic similarities for low-resource language performance, and code data to enhance reasoning capabilities.
2. Lawfulness (Licensing):
   • Allowed Licenses: Only data under CC-0, CC-BY (excluding NC/SA), and Public Domain is included.
   • Excluded Licenses: Data with Non-Commercial (NC), No-Derivatives (ND), or Share-Alike (SA) clauses (e.g., Wikipedia) are excluded to ensure the resulting LLM can be commercially deployed without license contamination.
   • Code: Restricted to permissive open-source licenses (Apache 2.0, MIT, BSD, Unlicense).
3. Harm & Bias: The team actively filters out data from unmoderated sources (like Reddit) and applies rigorous curation to minimize bias and harmful content.

B. Data Creation and Collection Strategies

The corpus is a hybrid of curated existing data and newly created collections:

• Targeted Crawls & Collaboration: Data was collected in collaboration with Dutch government bodies, archives, and research institutions (e.g., Openstate Foundation, National Archives). This included explicit consent for scraping, satisfying AI Act requirements.
• Digitalization: Old public domain texts (100+ years) were digitized using OCR and Handwritten Text Recognition (HTR), followed by strict filtering for low-quality scans.
• Synthetic Data (Type 1 Only):
  • Wikidata-Synth: Structured Wikidata triples (CC-0) were converted to Dutch sentences via code, then paraphrased by an LLM (Phi-4) to improve readability.
  • YouTube-Commons-Synth: English/Spanish transcripts were translated to Dutch using the MADLAD-400 model.
  • Note: Type 2 synthetic data (LLM-generated from scratch) was excluded to prevent hallucination and ensure traceable knowledge sources.
• Web-Crawl Filtering (C5): The authors utilized C5, a tool they developed with KU Leuven and INT, which uses structural HTML parsing (DOM analysis) rather than simple regex to detect Creative Commons licenses. For the GPT-NL corpus, C5 data underwent a strict "zero false positive" filter, verifying licenses manually on high-volume domains.

C. Evaluation and Curation Pipeline

A multi-stage pipeline ensures data quality before inclusion:

1. Normalization & Language Detection: Identifying and filtering non-target languages.
2. Heuristic Filtering: Removing duplicates, short lines, and low-quality text.
3. Manual Inspection: A representative sample is manually reviewed to tune filter thresholds (e.g., handling dialects vs. standard Dutch) and assess content quality.
4. Risk Analysis: Datasets are assigned a risk level based on potential bias, personal data leakage, or harmful content. High-risk datasets are down-sampled during training to mitigate negative model behavior.
5. PII Removal: Automated removal of personal identifiable information, verified via manual sampling.

3. Key Contributions

• The GPT-NL Public Corpus: The largest permissively licensed Dutch corpus, containing 36 billion Dutch tokens (21 unique collections) not found in other LLM pre-training sets.
• Multilingual & Code Expansion: Includes ~207B English, 232B Code, and 48B German/Danish tokens, all curated for compliance.
• C5 Tool: A novel structural license extraction tool that improves the accuracy of identifying permissive licenses in web crawls compared to regex-based methods.
• Legal Framework: A clear methodology for constructing LLM datasets that are legally safe for commercial use under EU law, explicitly addressing the "derivative work" debate by excluding NC/SA/ND licenses.
• Synthetic Data Pipeline: A validated approach for generating synthetic training data from structured, lawful sources (Wikidata) without introducing hallucinations.

4. Results

• Dataset Composition: The final corpus comprises 21 Dutch-only collections.
  • Sources: Includes government documents (municipal councils, parliament), judicial rulings (750k+ court cases), archival texts, and scientific publications.
  • Token Counts:
    • Dutch: ~36.4 Billion tokens.
    • English: ~207 Billion tokens.
    • Code: ~232 Billion tokens.
    • Other (German/Danish/Frisian): ~48 Billion tokens.
• Availability: The full dataset is publicly available on the Hugging Face Hub under a CC-BY license.
• Quality Control: The curation process successfully filtered out non-permissive content and reduced harmful bias while maintaining high linguistic diversity (including dialects).

5. Significance

• Commercial Viability: By strictly adhering to permissive licensing, this corpus enables the training of LLMs that can be legally deployed in commercial products within the EU, a significant barrier for many existing open-source models.
• Dutch Language Advancement: It provides a high-quality, Dutch-first foundation that reduces reliance on English-centric multilingual models, potentially improving performance in Dutch-specific tasks.
• Transparency & Reproducibility: The paper details the entire data lineage, from source to final token, setting a new standard for transparency in LLM data curation.
• Future-Proofing: The methodology addresses emerging EU regulations (AI Act, Copyright Directive), offering a blueprint for how researchers can build compliant datasets in an increasingly regulated AI landscape.

Limitations: The authors acknowledge the corpus is not all-encompassing (e.g., excludes Wikipedia due to CC-BY-SA) and lacks the "world knowledge" breadth of massive web crawls like FineWeb-2. Consequently, the full GPT-NL model training also utilizes proprietary, non-public datasets to fill these gaps. Additionally, some historical archival data contains harmful content, requiring users to exercise caution during sampling.