ArXiv-to-Model: A Practical Study of Scientific LM Training

1. Problem Statement

While frontier Large Language Models (LLMs) demonstrate impressive reasoning and mathematical capabilities, the practical engineering process of training domain-specialized scientific models from raw, open-access sources remains under-documented. Most reported systems rely on massive, proprietary, or curated corpora with undisclosed preprocessing pipelines.

Training directly from raw scientific sources (specifically arXiv LaTeX) introduces unique challenges:

• Heterogeneity: Multi-file project structures, custom macros, and inconsistent metadata.
• Symbolic Density: High concentration of mathematical notation and equations that standard tokenizers often fragment.
• Noise: Withdrawn papers, non-English text, and formatting artifacts.
• Resource Constraints: The lack of documented strategies for training scientific models under constrained compute budgets (e.g., single-node, dual-GPU setups).

2. Methodology

The authors present an end-to-end pipeline for training a 1.36B-parameter scientific language model (named KiteFish-A1-1.5B) from scratch using raw arXiv LaTeX sources.

A. Data Engineering Pipeline

The study utilized an 80GB text corpus derived from raw arXiv sources, processed through four stages:

1. Source Extraction & Validation: Downloading and verifying structural integrity of .tar archives.
2. Metadata Filtering:
   • Subject Focus: Restricted to Math, CS, Theoretical Physics (hep-th, hep-ph, quant-ph), and Statistics.
   • Temporal Filter: Papers published after 2000.
   • Quality Control: Removal of withdrawn manuscripts, short notes (<2,000 chars), and non-English text.
3. LaTeX Normalization: Stripping figures, references, and non-semantic commands while preserving mathematical expressions and structural environments.
4. Deduplication: Exact content hashing and near-duplicate detection to remove versioned re-uploads.
5. Data Mixture: A weighted sampling strategy up-sampling "Gold" scientific documents (2.0x weight) while retaining broader domain papers to prevent overfitting.

B. Tokenization Strategy

Recognizing that generic tokenizers fragment mathematical symbols, the authors evaluated custom BPE and SentencePiece tokenizers.

• Final Decision: Adopted a LLaMA-compatible SentencePiece tokenizer with a vocabulary size of ~102,400 tokens.
• Rationale: This choice ensured architectural compatibility, stable embedding initialization, and reduced the risk of token-ID misalignment, despite the potential benefits of a domain-specific tokenizer.
• Outcome: The pipeline yielded 52.18B tokens for pretraining and 5B tokens for post-training.

C. Model Architecture & Training Setup

• Architecture: Dense, decoder-only transformer (LLaMA-style) with 1.36B parameters.
  • Hidden dimension: 2048, Layers: 24, Heads: 16.
  • Untied input/output embeddings.
  • Rotary Position Embeddings (RoPE), SiLU activation, RMSNorm.
• Hardware: 2× NVIDIA A100 GPUs (80GB) with bfloat16 precision.
• Optimization: AdamW optimizer, ZeRO Stage 2 memory optimization, gradient accumulation, and activation checkpointing.
• Curriculum Strategy: A three-stage approach:
  1. Textual Warm-up: Abstracts/introductions only.
  2. Symbolic Integration: Full LaTeX bodies with theorems.
  3. Mixed Curriculum: Balanced prose and formulas.
• Scaling: The model was trained on 52.18B tokens (approx. 38 tokens/parameter), placing it in a data-rich regime relative to the Chinchilla optimal scaling law (which suggests ~20 tokens/parameter for compute efficiency).

3. Key Contributions

1. Reproducible Pipeline: A transparent, end-to-end guide for constructing a scientific corpus from raw arXiv LaTeX, detailing filtering heuristics and extraction failures.
2. Quantitative Yield Analysis: An analysis of how metadata filtering and extraction errors significantly impact usable token volume (often more than raw data availability).
3. Tokenizer Insights: Practical findings on tokenization for symbolic-heavy text, highlighting the trade-off between custom domain tokenizers and the stability of established architectures (LLaMA).
4. Training Dynamics under Constraints: Empirical data from 24 experimental runs analyzing stability, gradient behavior, and hardware bottlenecks on a dual-A100 setup.
5. Infrastructure Trade-offs: Identification of Storage/I/O throughput as a critical bottleneck that can rival compute capacity as a limiting factor in small-scale training.

4. Results

• Training Stability:
  • Small-Data Regime (20GB): Training was unstable, with oscillating loss and poor convergence (Run 24).
  • Full-Data Regime (200GB): Training exhibited smooth, monotonic convergence with reduced gradient noise (Runs 20 & 23).
• Performance:
  • Final validation perplexity: ~4.2 (exp(1.438)).
  • No significant divergence between training and validation loss, indicating no severe overfitting within the observed horizon.
  • GPU utilization sustained above 95% with no ECC errors or persistent I/O stalls.
• Capabilities: The model demonstrated strong familiarity with mathematical notation, LaTeX structures, and formal scientific writing. However, as a base model, it lacked instruction-following or conversational abilities without post-training alignment.

5. Significance and Limitations

Significance:

• Democratization: Proves that high-quality, domain-specialized scientific models can be trained on moderate hardware (2×A100), making scientific AI accessible to researchers without massive clusters.
• Engineering Focus: Shifts the narrative from "bigger models" to "better data engineering," showing that preprocessing decisions often have a larger impact on performance than architectural tweaks for small-to-mid-scale models.
• Transparency: Provides a rare, open account of the "messy" realities of scientific data cleaning, including failure modes and storage bottlenecks.

Limitations:

• Compute Constraints: Limited exploration of larger architectures or extended context windows (trained at 768 tokens despite 4096 support).
• Domain Bias: Restricted to specific scientific categories, limiting general-domain adaptability.
• Evaluation: Relies primarily on perplexity; lacks rigorous benchmarks for theorem proving or logical reasoning correctness.
• Reproducibility: The heavy reliance on specific storage throughput and large-scale LaTeX preprocessing pipelines may be difficult to replicate for researchers with limited infrastructure.

Conclusion

The paper concludes that for researchers operating under constrained budgets, careful data engineering and pipeline design are as critical as model architecture. The study offers a blueprint for building specialized scientific LMs, emphasizing that stable training in a data-rich regime is achievable even with limited compute, provided that storage I/O and preprocessing heuristics are rigorously managed.