Technical Summary: SCICODEPILE

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in general-purpose code generation, their capability to handle scientific code generation remains inadequately assessed. Existing resources suffer from two primary limitations:

1. Scale and Domain Coverage: Current benchmarks (e.g., SciCode, MatTools) are often too small for large-scale training or lack diversity in scientific domains. Conversely, large-scale datasets like ChemPile are often chemistry-centric and rely on noisy keyword-based filtering.
2. Evaluation Rigor: Existing benchmarks frequently lack executable verification. They often rely on textual similarity metrics (e.g., BLEU) rather than functional correctness, failing to capture the complex numerical logic, domain-specific semantics, and project-level dependencies required for reliable scientific software.

Consequently, the gap between current LLMs and the generation of reliable, runnable scientific code is not fully understood.

Methodology

The authors present SCICODEPILE, a comprehensive framework comprising a 128GB scientific code corpus and an executable benchmark. The construction pipeline consists of three tightly coupled components:

1. Repository Curation (Component 1)

Instead of simple keyword filtering, the authors employ a retrieve-then-filter pipeline:

• Vocabulary Expansion: Starting with 198 seed keywords across five major scientific domains (chemistry, life sciences, pharmaceutics, numerical methods, and AI-driven modeling), an LLM expands these into 213 queries including synonyms, abbreviations, and subtopics.
• Retrieval & Filtering: The pipeline queries GitHub to retrieve ~1.3 million candidates. It then applies conservative heuristics: excluding repositories with <10 stars, removing documentation-only projects, and using an LLM (Qwen3-32B) to verify that the README describes actual computational implementations rather than mere keyword mentions.
• Result: A curated pool of 37,737 high-quality repositories.

2. Multi-Granularity Data Alignment (Component 2)

The curated repositories are transformed into four complementary data formats to support different training and evaluation needs:

• F1 (Raw Code Files): 125GB of deduplicated code (3.26M files) spanning Python, C/C++, Fortran, Julia, and others, suitable for continued pretraining.
• F2 (README Summary): Structured summaries of repository intent, dependencies, and I/O conventions, providing project-level grounding.
• F3 (Function Instruction): 500,000 function-level instances pairing real code with natural language descriptions. These are generated using evidence-constrained prompts and verified by a domain-specialized model (NatureLM) for scientific relevance.
• F4 (Problem–Solution Pairs): 20,000 instruction-solution pairs derived from F3 snippets, designed to increase training signal diversity while remaining grounded in real code.

3. Executable Benchmark Construction (Component 3)

To address the lack of functional verification, the authors construct a benchmark of 200 tasks:

• Task Definition: Derived from F3, tasks require translating natural language scientific requirements into runnable Python code.
• Test Synthesis: For each task, an LLM (GPT-5.3-Codex) automatically synthesizes a minimal execution environment and a test harness. External dependencies are replaced with lightweight stubs to ensure sandbox compatibility.
• Verification: Models are evaluated on whether their generated code passes all test assertions in a sandboxed environment. Only instances where the reference implementation passes all tests are retained.

Key Contributions

1. SCICODEPILE Corpus: The largest scientific code corpus to date (128GB, 37,737 repositories), organized into four formats that explicitly link repository-level documentation with function-level evidence and task-level formulations.
2. Executable Benchmark: A 200-task benchmark equipped with sandboxed execution and automated test harnesses, enabling functional verification rather than just textual similarity.
3. Systematic Empirical Evaluation: A comprehensive study of 15 diverse LLMs (open-source and closed-source) across three tasks: prefix-to-suffix completion, fill-in-the-middle infilling, and instruction-following executable generation.
4. Training Utility Validation: Experiments demonstrating that continued pretraining on F1 and instruction tuning on F3/F4 significantly improve model performance on scientific tasks.

Experimental Results

The evaluation reveals that scientific code generation remains a significant challenge for current models:

• Code Completion (PTS & FIM): Even the best-performing models (Qwen2.5-Coder-7B-Instruct) achieve CodeBLEU scores of only 38.13 (prefix) and 38.37 (fill-in-the-middle). This indicates that models struggle to infer project-level conventions and domain-specific logic from incomplete context.

• Executable Generation: Performance drops drastically when functional correctness is required. The strongest model, GPT-5.4-mini, achieves only 12.30% Pass@1 on the executable benchmark. Most other models, including specialized code models like StarCoder2, score near 0%.

• Failure Patterns:

  • Instruction-tuned models tend to generate generic boilerplate or stop early, missing file-level coherence.
  • Base code models (e.g., StarCoder2) frequently fail due to syntax errors when asked to follow natural language instructions.
  • Advanced models often produce syntactically valid code that fails execution due to missing fields, incorrect logic, or runtime errors.

• Training Gains:

  • Continued pretraining of GPT-2 (124M) on F1 improved CodeBLEU by ×2.84.
  • Instruction tuning of Qwen2.5-Coder-0.5B on F3/F4 improved Pass@1 on the executable benchmark from 1.90% to 9.10% (×4.79).

Significance and Claims

The paper claims that SCICODEPILE provides the necessary infrastructure to systematically advance scientific code generation. By combining large-scale, domain-focused corpora with rigorous executable evaluation, the work:

• Quantifies the Gap: Demonstrates that current models are far from reliable scientific code generators, highlighting the difficulty of bridging syntactic validity with functional correctness in complex scientific domains.
• Enables Future Research: Offers a standardized resource for training and evaluating models on scientific tasks, moving beyond general coding benchmarks that lack domain specificity.
• Validates Training Signals: Confirms that the corpus provides effective training signals, as evidenced by the substantial performance gains in lightweight training experiments.

The authors conclude that future scientific code models must better leverage repository context and external dependencies to generate correct implementations, and that SCICODEPILE serves as a foundational step toward reliable AI assistants for scientific software development.