Technical Summary: Scientific Code Search at Scale

Problem Statement

Scientists increasingly rely on open-source tools to support research workflows, yet discovering relevant software among over 600 million GitHub repositories remains a significant bottleneck. Existing code search benchmarks (e.g., CodeSearchNet, CoSQA) focus on general software engineering tasks and fail to capture the domain-specific vocabulary, data formats (e.g., FITS, NetCDF), and instrument-specific needs of scientific computing. Current platforms like GitHub rely primarily on lexical matching rather than semantic understanding, leading to a "discoverability gap" where tools indexed under specific scientific terminology remain hidden from researchers using natural language queries. This gap results in duplicated effort, hinders reproducibility, and confines effective tools to their originating communities.

Methodology

The authors present a comprehensive framework for scientific code discovery, comprising a curated dataset and two novel retrieval benchmarks.

1. Data Collection and Curation

The authors constructed a high-quality corpus of 5,264 scientific repositories spanning five NASA Science Mission Directorate (SMD) divisions: Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences.

• Multi-Source Strategy: Repositories were aggregated from four sources: the NASA Earth Observations Knowledge Graph (mining DOIs from papers), curated GitHub organizations identified by domain experts, the Astrophysics Source Code Library (ASCL), and NASA's Science Discovery Engine.
• Quality Filtering: The pipeline normalized URLs, verified repository activity, and removed duplicates. Repositories were filtered to ensure they contained non-empty READMEs and were relevant to NASA science.
• Domain Classification: An LLM-based classifier (GPT-4.1-mini) was employed to assign each repository to a specific NASA SMD division, achieving F1 scores of 0.85 for Earth Science and 0.93 for Astrophysics.
• Context Enrichment: To address sparse documentation, the authors cleaned READMEs to remove boilerplate and crawled external links (e.g., academic papers, data repositories) to extract additional scientific context. This enriched representation includes cleaned READMEs, extracted topics, and crawled content.

2. Benchmark Construction

Two distinct retrieval benchmarks were introduced:

• Repository Search Benchmark: Designed to simulate the discovery of entire tools. It contains 219 expert-curated queries derived from real information needs of scientists in Earth Science, Astrophysics, and Planetary Science. Unlike automated benchmarks, these queries were formulated by subject matter experts (SMEs) starting from known repositories, resulting in multi-relevance annotations (average 2.3 ground-truth repositories per query).
• Code Snippet Retrieval Benchmark: Designed to locate specific implementations (functions/classes). It comprises 117,950 unique code snippets and 119,720 queries across seven programming languages (Python, C, C++, Java, JavaScript, Fortran, Matlab). The benchmark includes two query types:
  • Docstring-to-Code: Natural language descriptions of functionality.
  • Identifier-to-Code: Function or class names, testing the ability to map abbreviated, domain-specific naming conventions to implementations.

3. Evaluation

The authors evaluated baseline retrieval methods using standard Information Retrieval metrics (MRR, Recall@k, NDCG).

• Models: Comparisons included lexical baselines (BM25), general-purpose semantic models (all-MiniLM-L6-v2, Qwen3-Embedding-0.6B), domain-specific scientific text models (INDUS-Retriever, nasa-smd-ibm-st-v2), and code-specialized models (SFR-Embedding-Code-400M_R).
• Approaches: Evaluations covered single-stage lexical and dense retrieval, as well as hybrid methods (Reciprocal Rank Fusion and Cross-Encoder reranking).

Key Results

Repository Search

• Context Enrichment: Enriching repository representations with cleaned READMEs and crawled external context substantially improved retrieval performance across all methods and domains.
• Domain Variance: Performance varied significantly by domain. Astrophysics achieved the highest MRR@10 (0.87) due to standardized terminology (e.g., FITS, WCS). Planetary Science struggled the most (MRR@10 0.22) due to mission-specific vocabulary and sparse documentation.
• Model Performance: The domain-specific model (INDUS-Retriever) combined with a holistic text view (concatenating description, cleaned README, topics, and context) achieved the best results. Hybrid retrieval methods (combining lexical and semantic signals) outperformed single-stage baselines, improving Recall@10 by over 40% compared to BM25.

Code Snippet Retrieval

• General vs. Domain Models: Contrary to repository search, general-purpose LLM embeddings (Qwen3-Embedding-0.6B) outperformed domain-specific scientific text models. Qwen3 achieved an MRR@10 of 0.54, nearly double the lexical baseline (0.19), suggesting that strong general text understanding transfers effectively to scientific code retrieval without domain-specific fine-tuning.
• Query Type Disparity: A stark performance gap exists between query types. Docstring-based queries yielded strong results (MRR@10 ~0.76 for functions), while identifier-based queries were significantly more challenging (MRR@10 ~0.18–0.25). Lexical baselines (BM25) essentially failed on identifier queries (MRR@10 0.01) due to the lack of overlap between abbreviated code names and natural language queries.
• Language and Documentation: Performance varied by language, with C++ and C showing higher scores than Java or Matlab. Crucially, retrieval success was heavily dependent on documentation quality; domains with strict inline documentation standards (like Planetary Science's PDS standards) performed better at the snippet level despite having fewer repositories.

Significance and Claims

The paper claims to address a critical gap between existing code search benchmarks and the specific needs of the scientific community. Its primary contributions are:

1. First Scientific Code Benchmarks: It introduces the first benchmarks specifically designed for scientific code discovery, moving beyond general software engineering tasks.
2. Evidence of Documentation Culture: The results demonstrate that documentation culture (README quality vs. inline docstring quality) is a distinct and critical dimension for retrieval success, varying significantly across scientific domains.
3. Limitations of Current Models: The study highlights that while domain-specific pretraining helps document-level retrieval, it does not necessarily transfer to code-level tasks, where general-purpose LLM embeddings currently lead.
4. The Identifier Challenge: The paper identifies the retrieval of code via abbreviated identifiers as a major unsolved challenge, exposing how non-informative naming conventions in scientific codebases remain a barrier to discoverability.

The authors conclude that improving scientific code discovery requires advances on multiple fronts: richer documentation practices, domain-aware retrieval models, and benchmarks that capture the distinctive challenges of scientific software. All datasets and evaluation scripts are publicly released to support reproducible research.