SciML Agents: Write the Solver, Not the Solution

1. Problem Statement

Scientific Machine Learning (SciML) has traditionally focused on training neural networks (e.g., Physics-Informed Neural Networks, Neural ODEs) to directly predict solution values for scientific tasks. While promising, these approaches often struggle with achieving high accuracy and robustness across diverse physical regimes.

The authors propose a paradigm shift: instead of training models to learn the solution function, use Large Language Models (LLMs) as agents to write code that leverages decades of established numerical algorithms. The core challenge is not just syntactic code generation, but scientific appropriateness. An LLM must:

• Correctly interpret a natural-language description of a physical system (governed by Ordinary Differential Equations, ODEs).
• Analyze the mathematical properties of the ODE (e.g., stiffness, stability).
• Select the appropriate numerical solver (e.g., explicit Runge-Kutta for non-stiff vs. implicit BDF for stiff).
• Set correct parameters (tolerances, time steps) to ensure numerical validity.

Currently, there is no standard benchmark to evaluate whether LLMs can make these domain-specific numerical choices reliably.

2. Methodology

A. Benchmark Datasets

To address the lack of evaluation standards, the authors introduced two new datasets:

1. Symbolic-Equivalence Diagnostic Dataset:

   • Goal: Test if models rely on superficial pattern matching (e.g., "large coefficient = stiff") or perform genuine symbolic reasoning.
   • Construction: Contains "misleading" ODEs where the raw form suggests stiffness (due to large coefficients or complex trigonometric/inverse functions), but algebraic simplification reveals them to be non-stiff.
   • Subsets:
     • Trigonometric: Uses identities (e.g., $\sin(t+\pi/2) + \cos(t) = 0$).
     • Inverse Function: Uses inverse operators (e.g., $\arcsin(\sin(t)) = t$).
     • Algebraic: Uses expansion/factorization to cancel large terms.
   • Metric: Accuracy in selecting an explicit solver (the correct choice for the simplified non-stiff form).

2. ODE-1000 Benchmark:

   • Goal: Evaluate end-to-end code generation for diverse ODE tasks.
   • Construction: 1,000 diverse ODE problems derived from natural-language descriptions, paired with ground-truth Python solutions using scipy.integrate.solve_ivp.
   • Generation: Synthesized using GPT-4.1 to ensure diversity in stiffness, order, time intervals, and coefficients.
   • Verification: Code is executed in a sandbox; solutions are compared against analytical solutions (via SymPy) using Relative $L_2$ error. Only examples with error $< 0.01$ are retained.

B. Experimental Setup

• Models: Evaluated a wide range of open-source (Llama 2/3/3.1/3.3, Qwen 1-3) and closed-source (GPT-4.1) models, varying in size (0.6B to 405B parameters) and release date (2023–2025).
• Prompting Strategies:
  • Unguided (Base): Minimal instructions.
  • Guided: Explicit instructions to perform step-by-step reasoning, simplify algebraic expressions, and analyze coefficients before choosing a solver.
• Fine-Tuning: Supervised Fine-Tuning (SFT) was applied to smaller/older models using 900 examples from ODE-1000.

3. Key Contributions

1. New Benchmarks: Introduction of the Diagnostic Dataset (probing symbolic reasoning vs. pattern matching) and ODE-1000 (large-scale scientific code generation benchmark).
2. Evaluation Framework: A dual-axis evaluation measuring both code executability (syntax) and numerical validity (scientific accuracy against ground truth).
3. Empirical Analysis of Prompting vs. Fine-Tuning:
   • Demonstrated that guided prompting significantly boosts performance, particularly for newer instruction-following models.
   • Showed that fine-tuning is critical for older or smaller models to achieve high reliability, while large, newer models (e.g., Qwen3-8B) often achieve near-perfect performance with prompting alone.
4. Insight on "Thinking": Found that enabling "reasoning modes" (chain-of-thought) in newer models drastically improves their ability to handle adversarial, misleading problems.

4. Key Results

Diagnostic Dataset (Symbolic Reasoning)

• Pattern Matching Failure: Older models and smaller models frequently failed the diagnostic tasks, incorrectly selecting implicit (stiff) solvers due to large coefficients, failing to simplify the equations.
• Reasoning Success: Newer models (Qwen3, GPT-4.1) achieved high accuracy (often >90% with guided prompts).
• Impact of Guidance: Guided prompts improved accuracy significantly. For example, Qwen3-0.6B improved from ~58% to ~65% on trigonometric tasks with base prompts, but reached 100% with guided prompts and reasoning mode.
• Scale vs. Architecture: Newer generations often outperformed larger, older models (e.g., Qwen3-0.6B outperformed Llama3-70B on some tasks), suggesting architecture and training data quality matter more than raw parameter count for reasoning.

ODE-1000 Benchmark (Code Generation)

• Executability: Fine-tuning dramatically improved code execution rates for small models (e.g., Qwen3-0.6B execution rate jumped from 27% to 97%).
• Numerical Accuracy:
  • Fine-Tuning: Reduced mean relative $L_2$ error significantly (e.g., Qwen3-0.6B error dropped from 0.37 to 0.10).
  • Prompting: Newer large models (e.g., Qwen3-8B, GPT-4.1) achieved near 100% accuracy and low error rates using only guided prompting, without fine-tuning.
• Conclusion: For simple ODE tasks, a specialized SciML agent can be built either by fine-tuning small models or by carefully prompting large, newer models.

5. Significance and Future Work

• Paradigm Shift: The paper validates that LLMs can serve as effective "SciML Agents" that bridge natural language descriptions and mature numerical solvers, offering a more robust alternative to training neural networks from scratch for specific physical problems.
• Practicality: It establishes that for many scientific tasks, the "write the solver" approach is viable, provided the model is guided to reason about mathematical properties rather than just generating code syntax.
• Future Directions:
  • Extending benchmarks to Partial Differential Equations (PDEs), boundary value problems, and chaotic/stiff regimes.
  • Incorporating tool use for symbolic simplification and stability checks.
  • Developing property-based testing and invariants for scientific code verification.

Open Source: The authors have released the code, datasets, and benchmarks at https://github.com/SqueezeAILab/sciml-agent.