💬 NLP

CodePDE: An Inference Framework for LLM-driven PDE Solver Generation

This paper introduces CodePDE, the first inference framework that leverages large language models to generate PDE solvers as code, demonstrating that advanced inference strategies and test-time scaling enable LLMs to effectively solve complex partial differential equations while providing critical insights into their reasoning, debugging, and reliability trade-offs.

Original authors: Shanda Li, Tanya Marwah, Junhong Shen, Weiwei Sun, Andrej Risteski, Yiming Yang, Ameet Talwalkar

Published 2026-02-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shanda Li, Tanya Marwah, Junhong Shen, Weiwei Sun, Andrej Risteski, Yiming Yang, Ameet Talwalkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, complex puzzle representing the laws of physics—things like how heat spreads, how water flows, or how air moves around a wing. In the scientific world, these puzzles are called Partial Differential Equations (PDEs).

For decades, solving these puzzles has been like trying to fix a Swiss watch with a hammer. You need a master watchmaker (a human expert with deep math knowledge) to carefully assemble the gears (numerical methods). It takes a long time, costs a fortune, and if you make one tiny mistake, the whole thing breaks.

Recently, scientists tried using "neural networks" (a type of AI) to solve these puzzles. But think of these as black boxes. You feed them data, and they spit out an answer. You don't know how they got there, and if they get it wrong, you can't easily fix them. Plus, they need to be "fed" massive amounts of data before they can even start working.

Enter CodePDE, the new framework introduced in this paper.

The Big Idea: The "AI Architect"

Instead of asking the AI to guess the answer directly, CodePDE asks the AI to write the instruction manual for solving the puzzle.

Think of it this way:

Old Way: You ask a chef to cook a meal, but you don't know the recipe. If it tastes bad, you can't fix it.
Neural Network Way: You ask a robot chef who has memorized millions of recipes. It cooks fast, but if it burns the soup, you don't know why, and you can't tell it to "use less salt" without retraining the whole robot.
CodePDE Way: You ask a brilliant AI chef to write a recipe for you. Once it writes the recipe (the code), you (or a computer) can read it, check if the steps make sense, and if the chef made a mistake (like "add 50 cups of salt"), the AI can read the error message, realize "Oops, too much salt," and rewrite the recipe until it's perfect.

How CodePDE Works (The 5-Step Dance)

The paper describes a framework called CodePDE that acts like a highly organized project manager for this AI chef. Here is the process in simple terms:

The Brief (Task Specification): You tell the AI, "I need a solver for the Burgers Equation (a specific fluid dynamics puzzle)."
The Draft (Code Generation): The AI writes a Python script (the recipe) to solve it. It might try a "Finite Difference" method (like measuring steps on a grid) or a "Spectral" method (like using sound waves).
The Glitch Fix (Debugging): The computer runs the code. If it crashes (e.g., "Error: Division by Zero"), the AI reads the error message, says, "Ah, I divided by zero! Let me fix that," and rewrites the code. It does this automatically, over and over, until the code runs smoothly.
The Taste Test (Evaluation): The code runs the simulation. The system checks: "Is the answer close to the truth? Did it get more accurate when we used a finer grid?"
The Polish (Refinement): If the answer is good but not great, the AI looks at the results and says, "I used a simple method here; maybe I should use a more advanced one to get better precision." It rewrites the code again to improve it.

What Did They Find?

The researchers tested this on 16 different "super-smart" AI models (like GPT-4, Claude, and DeepSeek) and some classic puzzles (Advection, Burgers, Navier-Stokes, etc.). Here are the key takeaways:

AI Can Be a Great Architect: With the right tools (debugging and refining), these AIs can write solvers that are just as good as, or sometimes even better than, the ones written by human experts.
Mistakes Are Okay (If You Fix Them): If you just ask the AI to write the code once, it fails about 60% of the time. But if you let it "debug" itself a few times, it succeeds about 84% of the time. It's like a student who gets a bad grade, studies the feedback, and then aced the retake.
More Thinking = Better Results: If you let the AI try 32 different versions of the code and pick the best one (a technique called "Test-Time Scaling"), the quality goes up. It's like asking a team of architects to draw 32 designs and picking the best one.
The "Reliability vs. Complexity" Trade-off: Some AI models are very safe; they write simple, boring code that works but isn't very fancy. Others are adventurous and try complex, high-level math. The paper found that the best models know when to be safe and when to be adventurous.
The "Black Box" is Gone: Unlike neural networks, the code written by CodePDE is human-readable. If the AI makes a mistake, a human scientist can look at the code, see exactly where it went wrong, and fix it. This is a huge deal for trust.

The One Hiccup

The paper admits that the AI sometimes struggles with very tricky puzzles (like the Reaction-Diffusion equation). In these cases, the AI tries to use a complex math trick when a simple, known "analytical" solution exists. However, the paper shows that if a human gives the AI a tiny hint ("Hey, use this specific formula for the reaction part"), the AI instantly solves it perfectly.

Why Should You Care?

This is a step toward democratizing science.
Imagine a biologist who knows their field inside out but hates coding. With CodePDE, they could describe a physical problem in plain English, and the AI would generate a professional-grade, bug-free, and verified computer program to solve it.

It turns the difficult art of "writing scientific code" into a conversation. It doesn't replace the scientist; it gives them a super-powered assistant that writes the code, checks its own work, and explains its reasoning, allowing humans to focus on the big ideas rather than the tiny syntax errors.

In short: CodePDE teaches AI to not just guess the answer, but to build the machine that finds the answer, and then to check its own work until it's perfect.

1. Problem Statement

Partial Differential Equations (PDEs) are fundamental to modeling physical systems in science and engineering. However, solving them presents significant challenges:

Traditional Numerical Methods: Require deep domain expertise, meticulous manual tuning, and extensive debugging. They are computationally expensive, especially for high-dimensional problems.
Neural PDE Solvers (e.g., PINNs, Neural Operators): While data-driven, they require large training datasets, often lack interpretability (acting as "black boxes"), and struggle to generalize to equations outside their training distribution.
The Gap: There is a need for a method that automates the creation of robust, efficient, and interpretable solvers without requiring massive offline training datasets or expert manual coding.

The paper poses the central research question: Can Large Language Models (LLMs) generate effective and efficient solver code for PDEs?

2. Methodology: The CodePDE Framework

The authors introduce CodePDE, an inference framework that treats PDE solving as a code generation task. Instead of predicting the solution directly, the LLM generates executable Python code implementing numerical algorithms (e.g., Finite Difference, Spectral Methods).

The framework operates through a five-step iterative process:

Task Specification: The PDE (governing equation, domain, boundary/initial conditions) is formatted into a natural language prompt.
Code Generation: The LLM generates a complete solver implementation (e.g., a solver function) using Chain-of-Thought (CoT) prompting to reason through numerical schemes (spatial discretization, time integration).
Debugging: The generated code is executed. If it crashes (syntax errors, runtime exceptions, NaN/Inf values), the error trace is fed back to the LLM to autonomously diagnose and fix the issue. This repeats iteratively (up to 4 rounds).
Evaluation: Successful solvers are evaluated against ground-truth reference solutions using:
- Normalized Root Mean Squared Error (nRMSE): Accuracy metric.
- Convergence Test: Checks if error decreases as grid resolution increases (ensuring numerical stability).
- Execution Time: Computational efficiency.
Solver Refinement: The LLM is provided with the nRMSE and execution logs of the current best solvers. It analyzes bottlenecks and numerical instabilities to generate improved implementations, aiming to lower error and increase speed.

Key Technical Features:

Test-Time Scaling: Uses "Best-of-N" sampling (generating multiple candidates and selecting the best) to boost performance.
Modularity: Supports various LLMs (proprietary and open-weights) and integrates with agentic workflows.
Libraries: Encourages the use of modern libraries like PyTorch and JAX for GPU acceleration.

3. Key Contributions

First Inference Framework for PDE Solvers: CodePDE is the first framework to combine PDE domain knowledge with LLM self-improvement and test-time scaling capabilities specifically for solver generation.
Demonstration of LLM Capabilities: The study proves that with advanced inference techniques (debugging, refinement, scaling), LLMs can generate solvers competitive with or superior to hand-crafted numerical solvers and specialized neural network baselines.
Comprehensive Empirical Study: The paper provides the first large-scale analysis of LLM-generated PDE solvers, uncovering critical insights:
- Reliability vs. Sophistication Trade-off: Some models prioritize simple, robust schemes (low failure rate, lower accuracy), while others attempt complex methods (higher accuracy, higher risk of failure).
- Distinct Skills: The ability to generate code from scratch does not always correlate with the ability to refine code. Advanced reasoning models (e.g., o3, DeepSeek-R1) excel at generation but do not consistently outperform standard models in the refinement stage.
- Failure Modes: LLMs struggle with tasks requiring specific mathematical insights (e.g., exploiting analytical solutions for reaction terms in Reaction-Diffusion equations) unless guided.

4. Experimental Results

The framework was evaluated on 16 LLMs across 5 representative PDE families: Advection, Burgers, Reaction-Diffusion, Compressible Navier-Stokes (CNS), and Darcy Flow.

Performance vs. Baselines:
- CodePDE solvers (with refinement) outperformed hand-crafted reference solvers on 4 out of 5 tasks.
- On the Burgers Equation, Gemini 2.0 Flash achieved an nRMSE of $1.06 \times 10^{-4}$ (70% improvement over the reference solver's $3.55 \times 10^{-4}$ ).
- LLM-generated solvers generally outperformed standard Neural Network baselines (FNO, PINN, U-Net) which require training.
Impact of Inference Techniques (Ablation Study):
- Debugging: Increased the bug-free solver generation rate from 41% (single-shot) to 84% (with self-debugging). Frontier models reached >90%.
- Refinement: Consistently improved solution quality across all PDEs.
- Test-Time Scaling: Accuracy improved significantly with increased sampling budget (Best-of-N), showing diminishing returns after $N=16$ .
Convergence Analysis:
- Models like Gemini 2.5 Pro and o3 demonstrated the ability to generate diverse, high-order numerical methods while maintaining low failure rates.
- Simpler models often defaulted to first-order methods to ensure stability, highlighting the reliability-sophistication trade-off.
Failure Analysis:
- The Reaction-Diffusion equation remained the most challenging. LLMs failed to utilize the analytical solution for the reaction term, leading to higher errors. However, providing a "human-in-the-loop" hint regarding operator splitting reduced the error from $1.44 \times 10^{-1}$ to $1.69 \times 10^{-3}$ , surpassing the reference solver.

5. Significance and Future Impact

Democratization of Scientific Computing: CodePDE lowers the barrier to entry for solving complex PDEs, allowing domain scientists without deep numerical analysis or programming expertise to generate high-quality solvers via natural language.
Interpretability: Unlike neural solvers, CodePDE produces human-readable code. This transparency allows for expert review, debugging, and verification, which is crucial for high-stakes scientific applications.
New Paradigm for AI in Science: The work shifts the paradigm from "training a model to solve a specific equation" to "using an LLM to write a solver for any described equation."
Guidance for Agent Design: The findings suggest that effective scientific agents require a structured workflow combining automated debugging, iterative refinement, and test-time scaling, rather than relying solely on raw generation capabilities.

In conclusion, CodePDE demonstrates that LLMs, when equipped with the right inference-time strategies, can serve as powerful, reliable, and interpretable engines for scientific computing, potentially rivaling traditional expert-crafted solvers.