daVinci-LLM:Towards the Science of Pretraining

1. Problem Statement

The paper addresses a critical structural paradox in Large Language Model (LLM) research:

• Commercial Constraints: Organizations with the computational resources for large-scale pretraining operate under commercial pressures that prioritize rapid deployment over systematic exploration, leading to opaque training processes (e.g., undisclosed data compositions, mixture ratios, and negative results).
• Academic Limitations: Academic institutions possess research freedom but lack the infrastructure for billion-parameter pretraining at scale, making systematic, large-scale ablation studies structurally infeasible.
• The Gap: Consequently, the foundational pretraining phase—which determines a model's capability ceiling—is critically under-explored. Research has disproportionately focused on post-training (alignment/finetuning), yet evidence suggests post-training struggles to overcome capabilities established during pretraining.

2. Methodology

The authors propose daVinci-LLM, a project that occupies the intersection of industrial-scale resources and full research freedom. They adopt a fully-open paradigm, treating openness as a scientific methodology by releasing model weights, intermediate checkpoints, complete data pipelines, and 200+ controlled ablation studies.

A. Data Strategy: The Data Darwinism Framework

To address the lack of systematic data processing methodology, the authors utilize the Data Darwinism framework, a principled L0–L9 taxonomy ranging from basic acquisition to synthetic generation:

• L0–L2 (Acquisition & Filtering): Raw data collection, format normalization, and rule-based filtering (e.g., deduplication, language filtering).
• L3 (Model-Based Filtering): Using lightweight classifiers to score educational value and domain relevance.
• L4 (Generative Refinement): Using frontier LLMs to actively transform content (e.g., removing noise, reorganizing structure, repairing formatting) while preserving semantic integrity.
• L5 (Cognitive Completion): Using LLMs to synthesize new reasoning chains, making implicit logical steps explicit (e.g., converting dense scientific papers into pedagogical Q&A).
• Data Composition: The corpus spans General Web Text, Code, Science, and QA. The authors explicitly annotate every data source with its Darwin Level and release the processed datasets.

B. Training Recipe: Two-Stage Adaptive Curriculum

The model is a 3B-parameter decoder-only transformer (based on Qwen2 architecture) trained from random initialization on 8 trillion tokens using a two-stage curriculum:

1. Stage 1: General Foundation (6T tokens):
   • Focuses on broad linguistic fluency and foundational knowledge.
   • Uses a progressive global batch size scaling strategy.
   • Adaptive Adjustment: Starts with ~68% Common Crawl, then shifts to reduce web text and increase Code and Science proportions as general knowledge saturates.
2. Stage 2: Reasoning Enhancement (2T tokens):
   • Shifts focus to high-order reasoning.
   • Introduces large-scale, high-density Structured QA data (L5 level).
   • Progressive Intensification: Begins with a balanced mixture (30% QA, 30% Code, 30% Science, 10% Web) to prevent domain collapse, then intensifies to 70% QA in the final substage to maximize reasoning capabilities.

C. Evaluation Protocol

The authors evaluate checkpoints every 5,000 steps across 19 benchmarks covering General Knowledge, Code, and Science. They utilize both Perplexity-based (PPL) and Generative-based evaluation protocols to analyze how different training strategies affect model behavior under different inference conditions.

3. Key Contributions & Findings

Through 200+ controlled ablations, the paper establishes several empirical laws of pretraining:

• Processing Depth > Volume Scaling:
  • Advancing data processing from L2 (rules) to L3 (model filtering) yields modest gains.
  • L4 (Generative Refinement) provides substantial improvements in complex reasoning (e.g., +7.0 points on MATH) by cleaning structural noise and reinforcing logical flow.
  • L5 (Cognitive Completion/Synthesis) enables targeted capability steering. Synthetic QA data acts as a high-precision tool, significantly boosting reasoning in specific domains (Code or Science) based on the source domain.
• Adaptive Training Dynamics:
  • Differential Saturation: General knowledge capabilities saturate early (around 1T tokens), while reasoning capabilities (Code/Science) continue to grow.
  • Diminishing Returns of Proportion Adjustment: Simply rebalancing raw text domains (Web vs. Code vs. Science) hits a ceiling. Once standard corpora saturate, further growth requires a format shift from unstructured text to structured reasoning data (QA).
• Data Mixture Design (Balance vs. Intensification):
  • Internal Balance: In the reasoning-heavy Stage 2, an equal balance between Code and Science prevents over-specialization and catastrophic forgetting of general capabilities.
  • Stage-Dependent Tolerance: High concentrations of QA data (70%) cause performance collapse if introduced too early (Stage 2-1). However, once a balanced foundation is established (Stage 2-1), aggressive intensification (Stage 2-2) yields significant gains without collapse.
• Evaluation Validity:
  • The choice between PPL-based and Generative evaluation significantly impacts model rankings. Models trained heavily on QA data often show "ranking reversals," performing better in generative settings (where they must produce answers) than in PPL settings (where they only select probabilities), highlighting that QA training teaches behavioral formats, not just facts.

4. Results

The daVinci-LLM-3B model achieves state-of-the-art performance for its size class, demonstrating that systematic pretraining can outperform larger models:

• Overall Performance: Achieves an overall average score of 51.72, comparable to OLMo-3 7B (despite having less than half the parameters) and significantly outperforming other 3B models (LLaMa-3.2-3B, Qwen-2.5-3B).
• Reasoning Capabilities:
  • Math (MATH): 62.80 (exceeds OLMo-3 7B by >23 points).
  • Code: 55.99 average (matches OLMo-3 7B).
  • Science: 48.30 (exceeds OLMo-3 7B).
• General Knowledge: Maintains competence comparable to larger baselines, proving that the aggressive reasoning focus did not cause catastrophic forgetting.

5. Significance

• Scientific Transparency: The paper bridges the gap between industrial scale and academic rigor. By releasing the complete decision-making logic, negative results, and intermediate checkpoints, it transforms pretraining from an "intuition-led craft" into an "evidence-based discipline."
• Methodological Blueprint: It provides a reproducible framework (Data Darwinism + Adaptive Curriculum) for the community to build upon, offering a clear path for improving base models without requiring massive new compute resources, but rather through smarter data processing and scheduling.
• Community Impact: The release of 200+ ablation results allows researchers to understand why certain configurations work, enabling the accumulation of scientific knowledge in pretraining dynamics rather than just the release of model weights.