Technical Summary: MixMinMatch – Cross-Source Agreement for Multilingual Pretraining Datasets

Problem Statement

Building high-quality pretraining corpora for Large Language Models (LLMs) via web scraping is computationally expensive and resource-intensive. For non-English languages, multiple research groups often independently crawl and filter the web, resulting in significant redundancy. While this redundancy is traditionally viewed as inefficiency—wasting resources on duplicate content—the authors argue it represents an untapped quality signal.

The core problem addressed is the lack of a computationally efficient, model-free method to filter low-quality text in multilingual datasets. Existing approaches often rely on expensive model-based quality classifiers (requiring inference passes for every document) or language-specific heuristics that fail to generalize across diverse scripts and morphologies. Furthermore, standard deduplication removes near-duplicates to save compute but discards the valuable metadata regarding which sources retained specific content.

Methodology: MixMinMatch

The authors propose MixMinMatch, a three-stage pipeline that leverages the redundancy of existing web corpora as a quality filter. The method treats the independent filtering decisions of different research groups as an ensemble of "voters" on content quality.

1. Mix

The pipeline aggregates multiple existing public pretraining corpora for a target language (e.g., Arabic, Turkish, Hindi) while preserving the provenance of each document. Each document is stored as a tuple $(x, s)$, where $x$ is the text and $s$ denotes the originating dataset. Before aggregation, language-specific quality filters (e.g., length constraints, script consistency, n-gram repetition checks) are applied independently to each source to remove obvious low-quality content.

2. MinHash

The aggregated corpus undergoes standard near-duplicate detection using MinHash and Locality-Sensitive Hashing (LSH).

• Process: Documents are converted into sets of character $n$-gram shingles ($n=5$). MinHash signatures are computed, and documents colliding in at least one LSH band are treated as candidate pairs.
• Clustering: Pairs with an estimated Jaccard similarity $\geq 0.8$ are clustered using Union-Find.
• Deduplication: One representative document is retained per cluster.
• Key Insight: This step is standard practice in LLM pretraining to avoid training on duplicates. MixMinMatch utilizes the clusters produced by this step, not just the deduplicated output.

3. Match

This stage extracts the quality signal from the deduplication process.

• Cross-Source Agreement: For each near-duplicate cluster, the algorithm counts the number of unique source datasets ($s$) that contributed a document to that cluster.
• Filtering: The "Matched" subset retains only the representatives of clusters where $|S(C_i)| \geq 2$. In other words, a document is kept only if it was independently recovered by at least two different source pipelines.
• Efficiency: This operation is computationally free relative to the deduplication step. It involves only counting source labels within existing Union-Find structures, requiring no additional inference passes or model training.

Key Contributions

1. Systematic Analysis of Redundancy: The authors quantify cross-source overlap across major multilingual corpora (C4, CulturaX, HPLT 2.0, FineWeb-2, etc.), demonstrating that tens of billions of tokens are shared across independent pipelines. For example, C4 and CulturaX share over 9 billion tokens of near-duplicate content in Arabic.
2. MixMinMatch Algorithm: A novel method that reframes cross-source agreement as an ensemble filter. It extracts a quality signal "for free" from the standard MinHash deduplication infrastructure already used in pretraining.
3. New Corpora: The release of three new multilingual pretraining datasets (AraMix, TurMix, HinMix) for Arabic, Turkish, and Hindi. These include both full MinHash-deduplicated versions and smaller, high-quality "Matched" subsets with per-document source counts.
4. Empirical Validation: Experimental evidence showing that cross-source matching identifies high-quality content that outperforms single-source baselines.

Results

The authors evaluated the generated datasets by pretraining 1.46B parameter Llama-style models on a fixed token budget (29.36B tokens) and measuring performance on language-specific benchmark suites (FineTasks).

• Arabic: The AraMix-Matched subset achieved an aggregate score of 0.161, a 4.5% relative improvement over the best single-source baseline (ArabicWeb24, 0.154). The full AraMix-MinHash mixture (178B tokens) matched the baseline performance (0.153) while providing 4.3× more unique tokens.
• Turkish: TurMix-Matched achieved a score of 0.204, a 5.5% relative improvement over the FineWeb-2 Turkish baseline (0.194).
• Hindi: HinMix-MinHash showed the largest gain, improving over the CulturaX-Hindi baseline by 11.6% (0.089 vs. 0.080). The matched subset also outperformed the baseline by 8.1%.
• Ablation Studies: Removing the "best" single-source baseline from the mixture did not negate the gains. For instance, TurMix-Matched without FineWeb-2 still outperformed the FineWeb-2 baseline, confirming that the quality signal arises from the agreement among the remaining sources, not merely from inheriting the quality of a single superior dataset.
• Comparison to Model-Based Filtering: The matched subset outperformed FineWeb2-HQ, a corpus filtered using expensive model-based quality scoring, despite MixMinMatch being model-free.

Significance and Claims

The paper posits that redundancy in existing web corpora is a resource, not just waste. The authors claim that content independently retained by multiple filtering pipelines is a strong indicator of quality, analogous to inter-annotator agreement in crowdsourcing or ensemble methods in machine learning.

The primary significance of this work is the demonstration that cross-source agreement functions as a free, model-free quality filter. By treating the standard deduplication step (MinHash) as a mechanism to identify consensus, practitioners can construct high-quality pretraining datasets without the computational overhead of running quality classifiers on every document. The authors conclude that the research community can extract additional value from existing datasets by systematically combining them, rather than treating each new project as a ground-up crawl.

The paper remains modest regarding scale, noting that experiments were conducted on 1.46B parameter models to ensure direct comparability with the FineWeb-2 protocol, though they suggest that data quality effects typically persist or amplify at larger scales. They also acknowledge that cross-source matching is complementary to, rather than a replacement for, other data selection methods.