Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training

Technical Summary: Data Darwinism – Unlocking the Value of Scientific Data for Pre-training

Paper Title: Data Darwinism – Part I: Unlocking the Value of Scientific Data for Pre-training
Authors: Yiwei Qin, Zhen Huang, Tiantian Mi, et al. (SII-GAIR)
Date: February 8, 2026

---

1. The Problem: The "Learnability Gap" in Scientific Data

While the scaling laws of model architectures are well-understood, the methodology for transforming raw data into high-quality training corpora remains fragmented and lacks a systematic theoretical framework.

The authors identify a specific, critical challenge in the scientific domain: the learnability gap. Scientific literature is characterized by extreme conceptual density, implicit reasoning chains, and expert-oriented exposition (the "Expert-to-Expert" paradigm). Consequently, even though scientific data is information-rich, raw scientific text (unprocessed) provides negligible performance gains during pre-training because the content is too "opaque" for models to internalize the underlying logic.

2. Methodology: The Data Darwinism Framework

The paper introduces Data Darwinism, a ten-level hierarchical taxonomy (L0–L9) that organizes data processing as an evolutionary trajectory. The framework moves from simple selection to complex generation:

• L0–L3 (Selection/Filtering): Focuses on acquisition, normalization, rule-based filtering (deduplication, language detection), and lightweight model-based filtering (topic/quality classification).
• L4 (Generative Refinement): A shift from selection to transformation. Using medium-to-large LLMs, this stage purifies content by removing non-educational noise (metadata, OCR artifacts, navigation elements) and repairing structural defects (fragmented equations/tables) without adding new knowledge.
• L5 (Cognitive Completion): The most advanced stage implemented in this work. It uses frontier LLMs to bridge the cognitive gap by:
  1. Reasoning Reconstruction: Expanding implicit logical leaps into explicit, step-by-step derivations.
  2. Terminological Explication: Contextualizing jargon inline.
  3. Pedagogical Bridging: Grounding abstract theories in concrete analogies.

Experimental Setup:
To validate this, the authors constructed Darwin-Science, a 900B-token scientific corpus. They also developed daVinci-origin-3B/7B, base models trained from scratch on a 5.37T-token corpus that deliberately excluded all scientific content. This "clean-room" approach ensures that any performance gains observed during continued pre-training (CPT) are unambiguously attributed to the processed scientific data rather than prior knowledge.

3. Key Contributions

1. Conceptual Framework: The first systematic hierarchy (Data Darwinism) for categorizing data processing operations.
2. Darwin-Science Corpus: The largest open-source, hierarchically processed (L0–L5) scientific corpus (900B tokens).
3. Darwin-Science-Eval: A rigorous benchmark of 150K expert-level questions designed to evaluate deep domain comprehension rather than elementary science.
4. Transparent Base Models: Release of the daVinci-origin models to enable contamination-free research.

4. Key Results

The authors conducted 600B tokens of continued pre-training and found:

• Hierarchy Unlocks Value: Low-level processing (L0–L3) yielded negligible gains. However, L4 (Refinement) provided a +0.38 gain, and L5 (Cognitive Completion) provided a massive +0.98 gain, totaling a +1.36 cumulative improvement.
• Substantial Performance Gains: On 20+ benchmarks, the processed data improved the 3B model by +2.12 and the 7B model by +2.95. On domain-aligned evaluations (Darwin-Science-Eval), gains amplified significantly to +5.60 (3B) and +8.40 (7B).
• Scaling Laws of Data: Larger models (7B) derive disproportionately more value from high-complexity scientific data than smaller models (3B), suggesting model capacity is a prerequisite for utilizing dense data.
• No Saturation: Unlike many training regimes, the benefits of the processed scientific data showed no signs of diminishing returns throughout the 600B-token window.

5. Significance

This paper shifts the paradigm of data engineering from "filtering and cleaning" to "cognitive enrichment." It proves that for highly specialized domains, data quality is not a static attribute but a function of model-driven transformation. By providing a roadmap (the L0–L9 hierarchy) and the tools (Darwin-Science), the authors enable the development of next-generation AI systems capable of mastering complex, human-level scientific reasoning through a co-evolutionary process of data and model improvement.