Technical Summary: CODEDISTILLER

Problem Statement

Automated Scientific Discovery (ASD) systems aim to generate and execute code-based experiments to produce scientific insights. However, current systems face a critical limitation: they rely heavily on parametric knowledge (information acquired during model training) to generate code. This restricts their ability to perform experiments requiring highly specific methods, measurements, or protocols not explicitly present in the model's training data. Consequently, existing approaches are forced to either mutate a small number of manually crafted experiment examples or operate with limited quality and reach. There is a lack of scalable methods to convert the vast, unstructured code available in scientific repositories into vetted, functional examples that agents can reliably use.

Methodology: CODEDISTILLER

The authors introduce CODEDISTILLER, a system designed to automatically distill large collections of scientific GitHub repositories into a library of working, domain-specific code examples. The workflow consists of two primary phases:

1. Relevant File Classification

Given that repositories often contain hundreds of nested files, the system first performs static information gathering to identify files relevant to the repository's core functionality.

• Process: An LLM scans individual files, classifying them into coarse-grained types (code, documentation, scripts, data, other) and finer-grained categories (e.g., entry points, instructions, existing examples).
• Scoring: Each file is assigned a relevance score (1–5) and metadata regarding computational requirements (e.g., GPU needs) or critical configuration instructions.
• Goal: To filter the context window, providing the code generation model only the most task-relevant files to increase success probability.

2. Code Generation and Iterative Debugging

Using the highest-ranked files and the repository's core purpose, the system generates and debugs a working example.

• Generation: A base LLM generates four components: (1) executable Python code, (2) library requirements, (3) a Bash runscript with Conda environment setup, and (4) metadata (description, resource requirements, inclusion/exclusion criteria).
• Execution & Instrumentation: The code runs in an Ubuntu cloud container. The system captures outputs, including timestamped logs, JSON results, and raw STDOUT/STDERR.
• LLM-as-a-Judge Loop: An LLM evaluates the execution logs to determine if the code functioned correctly. If issues are detected, the system reflects on the logs and modifies the code. This loop repeats until success or until a limit (typically 8 iterations) is reached.
• Output: Successful examples are self-contained, ready-to-use scripts suitable for downstream ASD agents.

Key Contributions

1. CODEDISTILLER System: A novel pipeline for automatically converting specialized scientific GitHub repositories into debugged, working examples for automated discovery systems.
2. Materials Science Evaluation: An empirical evaluation on 250 materials science repositories, demonstrating that base models can successfully distill between 26% and 74% of repositories, depending on the model's capability and cost.
3. Downstream Impact: Evidence that augmenting an ASD agent (specifically CodeScientist) with a CODEDISTILLER-generated library results in more accurate, complete, and scientifically sound experiments compared to agents relying solely on general domain code examples.
4. Evaluation Metrics Analysis: A detailed characterization of the agreement between "LLM-as-a-Judge" and domain-expert ratings, finding moderate agreement in downstream A/B testing but variable agreement in the distillation task itself.

Results

Distillation Performance

The system was evaluated using three base models: GPT-OSS-120B, GPT-5, and Claude Sonnet 4.5.

• Success Rates: Automatic metrics (LLM-as-a-Judge) showed success rates ranging from 61.6% (GPT-OSS) to 75.6% (Claude). However, manual evaluation by a domain expert revealed significant overestimation by the automatic metrics.
  • GPT-OSS-120B: Reported 61.6% success; expert-validated correct functionality was only 25.9%.
  • GPT-5: Reported 70.4% success; expert-validated was 69.0%.
  • Claude Sonnet 4.5: Reported 75.6% success; expert-validated was 74.1%.
• Cost vs. Performance: Higher-performing models incurred significantly higher costs. The average cost for a successful example using Claude was 19 times higher than GPT-OSS.
• Runtime: Successful examples took 13–21 minutes on average, requiring 1.9–2.4 debug iterations. Unsuccessful attempts were costly, often hitting the 8-iteration limit.

Downstream Evaluation (A/B Testing)

The authors tested whether CODEDISTILLER examples improved the performance of the CodeScientist agent on 50 materials science discovery tasks.

• Preference: In A/B tests comparing a baseline agent (general examples) vs. an augmented agent (CODEDISTILLER library), the augmented system was preferred in >50% of cases across accuracy, completeness, and scientific soundness.
• Qualitative Improvements: The augmented agent produced solutions with better data grounding (using real datasets vs. synthetic), more physically accurate modeling (using domain-specific potentials vs. generic ones), and more canonical solutions (using established libraries vs. manual implementations).
• Metric Agreement: Cohen's Kappa between domain experts and LLM-as-a-Judge ratings for the downstream tasks ranged from 0.62 to 0.77 (moderate agreement). Accuracy showed the strongest agreement (0.77), while completeness showed the weakest (0.62).

Significance and Claims

The paper claims that CODEDISTILLER effectively addresses the bottleneck of manual code curation in automated scientific discovery. By automatically converting unstructured repository code into vetted examples, the system allows ASD agents to expand their capabilities beyond their parametric knowledge without manual effort.

The authors emphasize that while LLM-as-a-Judge metrics are useful for scaling evaluation, they can significantly overestimate performance compared to human experts, particularly for lower-cost models. However, the downstream results suggest that even with these limitations, integrating automatically distilled libraries significantly improves the scientific quality of agent-generated experiments. The work positions CODEDISTILLER not as a replacement for general-purpose agents, but as a specialized tool or preprocessing step that can be integrated into broader discovery workflows to enhance reliability and domain specificity.