Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection

1. Problem Statement

As Large Language Models (LLMs) scale, the focus of data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio through quality filtering. While model-based classifiers have proven effective for high-resource languages (e.g., English, French), a critical bottleneck exists for low-resource languages: there is often insufficient native high-quality data to train robust, standalone quality classifiers.

The paper addresses the following core questions:

• Can quality markers in embedding spaces show cross-lingual consistency, allowing high-resource languages to "subsidize" the filtering of low-resource ones?
• Can classifiers trained on one language family generalize to typologically distant languages?
• How can decision boundaries be refined to distinguish high-quality content from fluent but low-utility text?

2. Methodology

A. Dataset Construction (MKC-e)

The authors extended the FineWeb2-HQ framework by creating an extended positive anchor dataset (MKC-e).

• Positive Anchors: Combined original high-quality sources (Multilingual MMLU, Aya Dataset, OpenAssistant-2) with new instruction-based and synthetic sources:
  • Tagengo: Multilingual chat data (74 languages).
  • MURI-IT (Wikipedia Subset): Instruction-output pairs across 200 languages, filtered for encyclopedic prose.
  • EuroBlocks-SFT-Synthetic: Synthetic SFT data for EuroLLM (35 languages).
  • WikiQA: Real-world queries linked to Wikipedia answers (65 languages).
• Negative Anchors: Raw FineWeb2 corpus, assumed to contain boilerplate, informal prose, and noise.

B. Classifier Architecture & Training

• Embeddings: All text is encoded using XLM-RoBERTa (768-dimensional embeddings).
• Model: A simple Multi-Layer Perceptron (MLP) with a single hidden layer (256 dim, ReLU, 20% dropout) and sigmoid output.
• Sampling Strategy:
  • Balancing: 100,000 positive documents per language (upsampled by max 3x for low-resource languages to counter imbalance).
  • Negative Sampling: Two strategies were compared:
    1. Random Sampling: Standard uniform selection from FineWeb2.
    2. Q3 Sampling (Hard Negatives): Selecting documents scoring in the 50th–75th percentile of a preliminary classifier. These represent "fluent but low-utility" text (e.g., repetitive procedural content), forcing the model to learn subtler distinctions beyond surface fluency.

C. Evaluation Protocol

• Downstream Task: Training a 1B parameter Apertus model on filtered data (103B tokens).
• Metrics:
  • Normalized Accuracy: Average performance across diverse benchmarks (ARC, MMLU, XNLI, Belebele, etc.).
  • Average Rank: Stability metric across benchmarks.
• Languages Tested: French (Romance, High-Resource), Spanish (Romance, High-Resource), Arabic (Semitic, Morphologically Complex), Chinese (Sino-Tibetan, Logographic).

3. Key Contributions

1. Multilingual Synergy: Demonstrated that massive multilingual pooling frequently outperforms monolingual baselines. A classifier trained on multiple languages achieves higher rank stability and aggregate accuracy than a classifier trained on a single language, even for high-resource languages like French.
2. Empirical Validation of Cross-Family Transfer: Proved that classifiers trained on typologically distant language families (e.g., Nordic languages: Swedish, Danish, Norwegian) can effectively filter high-quality content in French.
   • Surprising Finding: The Nordic classifier outperformed the native French HQ baseline in aggregate accuracy, suggesting that quality signals (logical coherence, information density) are abstract and transferable across language families, rather than relying on surface-level syntax.
3. Q3 Sampling Strategy: Introduced a Third-Quartile (Q3) negative sampling technique. By training against fluent but low-utility text rather than random noise, the decision boundary is sharpened, significantly improving performance in high-resource settings.
4. Retention Rate Tuning: Showed that the standard 10% retention rate (from prior work) is often too aggressive. Tuning retention rates (e.g., to 15% for French) yields better downstream performance, indicating that aggressive filtering can discard useful high-quality content.

4. Key Results

• Multilingual vs. Monolingual:
  • Chinese: The Multilingual (ML) classifier achieved the highest aggregate accuracy (0.4236) and best average rank (1.31), outperforming the monolingual HQ baseline.
  • Spanish: ML strategy achieved an aggregate accuracy of 0.3855 vs. 0.3753 for HQ.
  • French: While HQ was strong, the ML (Q3) strategy achieved the best overall performance (0.4147 aggregate accuracy), combining multilingual signals with hard negative sampling.
• Cross-Family Transfer (The "Nordic" Experiment):
  • A classifier trained only on Nordic languages achieved an aggregate accuracy of 0.4151 on French data, slightly outperforming the native French HQ baseline (0.4089).
  • This confirms that quality features (e.g., Wikipedia formatting, instruction templates, logical structure) are shared across the embedding space regardless of linguistic distance.
• Q3 Impact:
  • In French, using Q3 negatives improved the monolingual classifier's accuracy from 0.4089 (HQ) to 0.4167.
  • In Spanish, the ML (Q3) strategy yielded the best results (0.3887), surpassing standard ML filtering.
• Retention Rate: Increasing retention from 10% to 15% for French improved aggregate accuracy from 0.4064 to 0.4209. Conversely, for Arabic, aggressive filtering (10-20% retention) degraded performance compared to the standard 56% retention.

5. Significance and Conclusion

This work fundamentally shifts the paradigm of multilingual data curation:

• Democratization: It provides a viable path for training high-quality classifiers for low-resource languages by leveraging data from high-resource, typologically distant languages.
• Robustness: The study reveals that "quality" in embedding space is a shared manifold of logical coherence and information density, not just linguistic fluency.
• Practical Guidelines:
  • Do not rely solely on monolingual data for high-resource languages; multilingual pooling offers stability and performance gains.
  • Refine negative sampling: Use Q3 (hard negatives) to distinguish high-quality content from "fluff."
  • Tune hyperparameters: Retention rates must be language-specific; a one-size-fits-all approach (e.g., 10%) is suboptimal.

The paper concludes that while scale alone does not guarantee stability, multilingual pooling combined with refined sampling strategies (Q3) creates the most robust general-purpose quality classifier, effectively bridging the gap between high and low-resource language curation.