💻 computer science

Executable Chain-of-Thought Symbolic Regression with Large Language Models for Automated Physical Discovery

This paper introduces \ours, a verification-carrying framework that transforms large language model-based symbolic regression into a rigorous process by executing chain-of-thought reasoning as a typed Symbolic Derivation Graph, enforcing hard constraints and soft verification signals to produce compact, machine-checkable physical laws with executable derivation traces.

Original authors: Lingyi Li

Published 2026-10-08

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Lingyi Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Science often feels like a process of compression. A scientist observes a vast collection of events—planets moving across the sky, chemicals reacting in a flask, or water flowing through a pipe—and seeks to distill that chaos into a single, compact mathematical law. This law is not merely a tool for predicting the next data point; it is a statement of how the world works, one that can be inspected, understood, and extended to new situations. For centuries, finding these laws has required human intuition, but the sheer volume of modern data has made the search for them increasingly difficult. The space of possible equations is so vast that even powerful computers struggle to find the right one without getting lost in a sea of incorrect possibilities. The challenge is not just to find a formula that fits the data we have, but to find the specific formula that remains true when the conditions change, when the data is noisy, or when we look at a part of the system we have never seen before.

In recent years, researchers have turned to large language models—systems trained on massive amounts of text and code—to help with this search. These models are excellent at recalling familiar patterns and suggesting plausible structures for equations. However, a significant gap has remained between what these models can propose and what they can prove. A model might suggest a formula that looks correct on the surface, but the steps it took to get there might be logically flawed, physically impossible, or mathematically unstable. It is like a student who writes down the right answer on a test but cannot show the work, or worse, shows work that contains hidden errors that only become apparent when the problem is slightly changed. The existing methods often treated the reasoning process as a free-form conversation, where the final equation was the only thing that mattered, leaving the path to that equation unverified and potentially unreliable.

To address this, the author of this study have developed a new framework called CoT-SR. This system changes the way computers search for scientific laws by treating the reasoning process itself as a rigorous, step-by-step construction rather than a guess-and-check game. Instead of letting a language model write a paragraph of text to justify an equation, the system forces the model to propose specific, executable actions that transform a simple starting expression into a more complex one. Each step is a typed action, such as adding a new term, changing a variable, or applying a known physical rule. Crucially, every single step is immediately checked by a separate, independent system to ensure it is valid. The system verifies that the math is syntactically correct, that the physical units make sense (for example, ensuring you are not adding meters to seconds), and that the equation behaves reasonably under extreme conditions.

The core innovation of this work is the creation of a "symbolic derivation graph." Imagine a map where every path represents a different way to build an equation. In previous methods, the computer might jump from one idea to another based on a vague hunch. In this new approach, the computer can only move along a path if the next step passes a strict set of rules. If a proposed step violates the laws of physics or the rules of algebra, the path is immediately closed off, and the system never wastes time exploring that dead end. This process turns the search for a law into a journey through a landscape of valid possibilities, where every move is documented and verified. The system keeps a record of every successful step and every failed attempt, creating a complete, auditable trail that explains exactly how the final equation was derived and why alternative paths were rejected.

The researchers tested this system on a wide variety of challenges, ranging from classic physics problems involving planetary motion and thermodynamics to complex, synthetic datasets designed to trick artificial intelligence. They found that CoT-SR was significantly better at finding the correct equations than previous methods. In tests where the goal was to recover a known physical law exactly, the new system achieved the highest exact-recovery rate among the compared methods, as detailed in the benchmark results, outperforming the next best approach. More importantly, the system proved much more robust when faced with noisy data or when asked to predict behavior outside the range of the original measurements. While other methods often failed when the data was slightly distorted or when the input variables were stretched to new extremes, CoT-SR maintained its accuracy. This is because the system prioritizes equations that are structurally sound and physically consistent, rather than just those that happen to fit the current numbers perfectly.

A key feature of the new framework is its ability to actively test its own hypotheses. When the system is unsure between two competing equations, it does not simply pick the one that looks better on paper. Instead, it calculates where the two equations would disagree the most and asks for a new measurement at that specific point. This is similar to a scientist designing a critical experiment to distinguish between two theories. By focusing its limited resources on the most informative data points, the system can quickly rule out incorrect possibilities and zero in on the true law. In simulations, this active approach allowed the system to eliminate wrong answers much faster than if it had just collected data randomly. The result is not just a final equation, but a complete package that includes the equation, the step-by-step derivation, and a record of the evidence that supports it.

The study also highlights the limitations of relying solely on the memory of a language model. Many previous systems struggled when presented with equations that had been slightly altered or disguised, because they were simply recalling patterns from their training data rather than truly understanding the underlying logic. CoT-SR, by contrast, relies on the validity of the derivation steps. Even if the model has never seen a specific equation before, it can construct it correctly if the logical steps are sound. This suggests that for scientific discovery, the ability to verify the process is more important than the ability to recall the answer. The researchers found that when they removed the strict verification steps, the system's performance dropped significantly, confirming that the rigorous checking of every step is what makes the approach effective.

Ultimately, this work represents a shift in how artificial intelligence can assist in scientific discovery. It moves away from the idea of a black box that outputs a prediction and toward a transparent system that builds knowledge step-by-step, with every claim backed by evidence. The final output is not just a formula, but a verified story of how that formula was found. This distinction is vital for science, where understanding the "why" is often as important as the "what." By ensuring that every intermediate step is executable and checkable, the researchers have created a tool that can help scientists navigate the complex search for new laws, offering a level of reliability and clarity that was previously out of reach. The system does not claim to be infallible, and it still requires human oversight, but it provides a powerful new way to combine the pattern-recognition strength of language models with the rigorous demands of physical science.

Technical Summary: Executable Chain-of-Thought Symbolic Regression with Large Language Models for Automated Physical Discovery

1. Problem Formulation

Symbolic Regression (SR) aims to discover compact analytic laws that explain observational data. While modern systems have expanded the scale of SR using genetic programming, sparse identification, and neural sequence models, significant challenges remain. The hypothesis space grows super-exponentially with expression depth, and many algebraically distinct trees are observationally indistinguishable on finite samples. Furthermore, low in-domain error does not guarantee a physically meaningful law.

Recent approaches utilizing Large Language Models (LLMs) have injected stronger priors into SR by leveraging scientific and programming knowledge. However, existing LLM-based SR methods typically treat Chain-of-Thought (CoT) as an untyped, natural-language scratchpad. This creates three critical gaps:

Plausibility vs. Validity: An LLM may propose a familiar physical form that fails on transformed or anti-memorization benchmarks because intermediate statements are not formal transformations.
Fit vs. Identifiability: Multiple candidates may interpolate the same samples but disagree under dimensional rescaling, symmetry operations, or asymptotic limits.
Terminal vs. Process Verification: Verification is usually applied only after generation, failing to reshape the search distribution when intermediate steps violate constraints.

The core problem addressed is how to define symbolic regression as inference over a sequence of typed derivations where intermediate states are executable, falsifiable, and reusable, rather than relying on unsupported natural-language rationales.

2. Methodology: CoT-SR

The author proposes CoT-SR, a framework that operationalizes chain-of-thought as an Executable Symbolic Derivation Graph (SDG). Instead of generating a final equation directly, the system searches over derivation trajectories $\tau = (h_0, a_1, h_1, \dots, a_T, h_T)$ , where $h_t$ are symbolic hypotheses and $a_t$ are typed transformations.

2.1 Symbolic Derivation Graph (SDG)

The SDG is a directed graph where nodes are canonicalized expressions with optimized constants and physical types. Edges represent executable transformations from a set of six families:

$a_{inst}$ : Instantiation
$a_{comp}$ : Composition
$a_{nondim}$ : Non-dimensionalization
$a_{sym}$ : Symmetry rewriting
$a_{limit}$ : Limit rewriting
$a_{res}$ : Residual factorization

The LLM acts as a proposal prior, suggesting edges $(a_t, \eta_t)$ based on a compressed reasoning state $s_t$ . Crucially, the LLM does not commit to a final law; it proposes actions which are then instantiated by a symbolic executor and numerically optimized.

2.2 Typed Verification Obligations

The framework distinguishes between exact admissibility constraints (hard filters) and empirical evidence scores (soft ranking signals):

Exact Obligations: Syntax validity and dimensional typing. If units are provided, a typed grammar enforces dimensional consistency (e.g., $\log$ and $\exp$ require dimensionless inputs). An edge failing these checks receives zero posterior mass.
Empirical Evidence: Symmetry equivariance, limiting behavior, numerical stability, and residual utility. These are scored (e.g., $v_{sym}, v_{lim}, v_{stab}$ ) to rank admissible trajectories but do not constitute formal proofs.

2.3 Verification-Weighted Trajectory Posterior

The search objective is defined over the entire trajectory rather than independent equations. The posterior probability is:
$p(\tau | D, \kappa) \propto \exp\left( \sum \log q_\theta(a_t, \eta_t | s_{t-1}, \kappa) - E(\tau) \right)$
where the energy function $E(\tau)$ includes data fit, description length (complexity of the final law and edges), and a penalty for violating verification evidence. This unifies the LLM proposal prior with data fit and scientific constraints in a single search objective.

2.4 Active Falsification

To resolve ambiguity where multiple symbolic laws fit the data equally well, CoT-SR employs an active falsification loop. The system identifies probe locations $x^*$ where high-posterior symbolic equivalence classes exhibit maximal predictive disagreement. This capability is conditional on the availability of an oracle, simulator, or experimental interface. When available, the system queries these sources at $x^*$ to update the posterior, suppressing or falsifying incompatible mechanisms. This transforms SR from passive curve fitting into iterative hypothesis discrimination for tasks where such interactive environments exist.

3. Key Contributions

Executable Chain-of-Thought: The author formulates SR as inference over a typed Symbolic Derivation Graph, replacing free-form rationales with algebraic actions that have machine-checkable preconditions and postconditions.
Verification-Weighted Trajectory Posterior: A unified search objective that combines data fit, complexity, and explicit verification signals (dimensional, symmetry, stability) rather than applying ad hoc filters post-generation.
Active Counterexample Acquisition: A mechanism to propose new measurement locations that maximize disagreement among surviving hypotheses, enabling iterative scientific falsification in settings where an oracle or simulator is available.
Comprehensive Benchmark Protocol: Evaluation across Feynman, SRBench, SRSD, Nguyen, and LLM-SRBench, with diagnostics designed to separate memorization, interpolation, extrapolation, and physical validity.

4. Experimental Results

The evaluation spans multiple benchmarks, including exact recovery, out-of-distribution (OOD) extrapolation, noise robustness, and search efficiency.

Exact Recovery: CoT-SR achieves state-of-the-art performance, with a mean exact recovery rate of 79.2% across benchmarks, significantly outperforming baselines like PySR (55.0%), PhySO (62.4%), and LLM-SR (69.2%). The gains are most pronounced on anti-memorization tasks (LLM-SRBench: 61.9% vs. 42.0%).
Complexity: CoT-SR produces more compact expressions, with an average operator count of 18.6, compared to 23.4 for LLM-SR and 31.2 for PySR.
OOD Generalization: On extrapolation tasks, CoT-SR achieves an OOD $R^2$ of 0.871, outperforming LLM-SR (0.781) and PhySO (0.721), demonstrating better adherence to physical laws rather than mere interpolation.
Noise Robustness: Under 10% Gaussian noise, CoT-SR maintains a 43% exact recovery rate, compared to 25% for LLM-SR and 18% for PySR.
Search Efficiency: CoT-SR requires fewer unique symbolic hypotheses (315) and LLM calls (61) to achieve high recovery compared to baselines, attributed to early pruning of invalid branches via hard obligations.
Active Falsification: On benchmarks with a synthetic oracle, the proposed disagreement-guided probe selection outperforms random sampling and uncertainty sampling, effectively concentrating posterior mass on the correct equivalence class. This module is evaluated specifically on tasks where oracle queries are permitted.
Ablation Studies: Removing the executable derivation trace causes the largest performance drop (from 81.2% to 70.4%), confirming the necessity of process-level verification. Dimensional verification and active falsification also provide significant, distinct improvements.

5. Significance and Claims

The paper claims that CoT-SR represents a shift from treating LLMs as oracles of scientific truth to treating them as priors over symbolic actions. The scientific credibility of the output is derived from the executable environment and the explicitly typed evidence accumulated along the derivation.

Key distinctions emphasized by the author include:

Process vs. Terminal Verification: Unlike "LLM + SR + Physics Filters" where checks are applied only at the end, CoT-SR integrates verification into the search trajectory. This allows the system to use the location and type of a failed transformation to redirect the search, rather than simply discarding a final candidate.
Auditable Outputs: The final output is not just an equation but a "verification-carrying" artifact containing the equation, an executable derivation trace, and a machine-checkable evidence record. This allows scientists to inspect exactly which constraints (e.g., dimensional, asymptotic) ruled out specific alternatives.
Modesty on Claims: The author explicitly notes that numerical checks (symmetry, stability) are empirical evidence, not formal proofs. They acknowledge limitations, such as the reliance on supplied metadata (units, symmetries) and the current focus on closed-form scalar relationships rather than PDEs or stochastic laws. Crucially, the active falsification component is limited to scenarios where an oracle or simulator is available to provide discriminative measurements.

The work suggests that for automated science, separating generative priors (LLMs) from machine-checkable constraints and falsification evidence is essential for producing robust, physically meaningful discoveries.