🤖 machine learning

InsightSR: Refining Symbolic Regression Search Spaces via Parallel Semantic and Structural LLM Guidance

InsightSR is a novel symbolic regression framework that enhances the PySR genetic programming engine by leveraging Large Language Models to iteratively refine the search space through semantic and structural guidance, achieving state-of-the-art accuracy and generalization by transforming deep expression tree construction into the assembly of shallow trees over semantically informed features.

Original authors: Yating Ling, Wenjing Cun, Zhitang Chen

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yating Ling, Wenjing Cun, Zhitang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Science has always relied on a simple, powerful idea: that the complex behavior of the natural world can be described by concise mathematical laws. When a physicist looks at a falling apple or a planetary orbit, they are not just seeing motion; they are looking for the hidden equation that governs it. This search for the underlying formula is called symbolic regression. Unlike standard computer models that learn patterns by adjusting millions of internal knobs to fit data, symbolic regression attempts to write out the actual equation itself, using familiar mathematical building blocks like addition, multiplication, and trigonometric functions. The goal is to find a rule that is not only accurate but also simple enough for a human to read and understand. However, finding these rules is incredibly difficult. The number of possible mathematical combinations grows so fast that it becomes a vast, chaotic ocean of possibilities. Most computer searches get lost in this ocean, producing formulas that fit the data perfectly but make no physical sense, or they simply fail to find the true law because the search space is too huge to explore completely.

A team of researchers has developed a new approach to navigate this chaos, combining the raw searching power of evolutionary algorithms with the reasoning ability of large language models. Their system, named InsightSR, does not ask the computer to guess the final answer directly. Instead, it uses the language model as a guide to reshape the search itself. Imagine a team of explorers trying to find a specific path through a dense, uncharted forest. In the old method, the explorers would wander randomly, hoping to stumble upon the right trail. In this new method, the language model acts as a seasoned guide who knows the general terrain. It does not walk the path for them, but it tells them which directions are physically impossible to take and suggests which tools might be useful for the journey.

The system works by splitting the guidance into two complementary streams. The first stream focuses on the "skeleton" of the equation. The language model looks at the physical units of the data—such as whether a variable represents time, distance, or mass—and proposes basic structures that must be dimensionally consistent. This means it rules out any mathematical combination that would violate the laws of physics, such as adding a time measurement to a distance. By seeding the search with these physically plausible starting points, the system avoids wasting time on billions of impossible formulas. The second stream focuses on the ingredients themselves. The language model suggests new ways to transform the raw data, such as squaring a variable or taking its sine, based on patterns it has seen in previous attempts. These new features are added to the pool of ingredients available to the search engine. Over time, the pool of ingredients grows richer, allowing the system to build complex relationships using simple, shallow combinations rather than trying to construct deep, tangled trees of raw data.

This process is not a one-time guess but a continuous loop of refinement. After the computer generates a set of candidate equations, the language model evaluates them. It checks not just how well they fit the numbers, but also how useful the new features were and whether the structure makes sense. This feedback is stored in a dynamic knowledge base that informs the next round of searching. The system learns from its own successes and failures, gradually narrowing the search toward the most promising solutions. This creates a self-correcting cycle where the search becomes more focused and efficient with every iteration.

The researchers tested this method on three distinct sets of challenges. First, they used a benchmark of 100 famous physics equations, ranging from classical mechanics to quantum theory. In this test, the system successfully recovered the exact original formula 95% of the time, a significant improvement over previous methods. It also performed exceptionally well on a broader set of scientific problems spanning chemistry, biology, and material science, achieving an accuracy of over 80% on tasks involving complex transformations. Finally, the team tested the system on real-world data, such as the vibrations of an oscillator and the growth patterns of bacteria. In these scenarios, the system not only found accurate formulas but also maintained its performance when tested on data it had never seen before, demonstrating a strong ability to generalize.

The results suggest that by embedding a language model as a guiding layer around a traditional search engine, it is possible to overcome the twin challenges of the vast search space and the need for physical consistency. The system does not replace the evolutionary search; it refines it, turning a blind, random exploration into a purposeful, guided discovery. By shifting the burden from constructing deep, complex trees of raw variables to assembling simple combinations from a rich, pre-enriched set of features, the method makes the discovery of scientific laws more efficient and reliable. This approach offers a practical path forward for automated scientific discovery, showing that the combination of human-like reasoning and machine-based searching can reveal the hidden mathematical laws that govern our world.

Technical Summary: InsightSR

Problem Statement

Symbolic Regression (SR) aims to discover parsimonious, interpretable mathematical laws directly from observational data. However, the field faces two primary challenges:

Combinatorial Explosion: The search space of candidate expressions grows exponentially with complexity, making the optimization problem inherently NP-hard.
Physical Consistency: Purely data-driven methods often yield expressions with high numerical accuracy that violate fundamental physical principles (e.g., dimensional consistency) or lack the parsimony required for scientific interpretation.

Existing approaches, including Genetic Programming (GP), Monte Carlo Tree Search (MCTS), and neural-symbolic methods (e.g., Deep Reinforcement Learning, Transformers), struggle with low sample efficiency, overfitting under noise, and a lack of dynamic reasoning regarding physical constraints. While recent attempts have integrated Large Language Models (LLMs) to leverage scientific priors, they are often limited by high inference overhead and the use of LLM guidance along only a single dimension (either structural generation or feature transformation), failing to exploit the synergy between the two.

Methodology: InsightSR Framework

InsightSR addresses these limitations by embedding LLMs as a guiding layer around the PySR genetic programming engine. Rather than relying on the LLM to generate final expressions directly, InsightSR uses the LLM to progressively transform the search space itself through a four-phase iterative loop:

1. Contextual Initialization and Unit Synthesis

The framework first resolves the physical dimensions of variables. It extracts dimensional information from metadata and problem descriptions. For ambiguous variables, an LLM infers probable dimensions based on semantic context. These unit assignments persist as hard constraints throughout the discovery loop, ensuring dimensional homogeneity and pruning physically inconsistent candidates early.

2. Parallel Guidance Strategy

The core innovation is a dual-pathway approach where the LLM simultaneously guides the search in two complementary dimensions:

Semantic Seed Pathway (Top-Down): The LLM proposes dimensionally consistent functional skeletons based on problem context and unit constraints. These skeletons serve as "physics-informed seeds" for the evolutionary search. The LLM audits candidate topologies for dimensional consistency before they enter the population and provides per-operator complexity biases to steer the GP engine toward physically plausible operator combinations.
Structural Feature Pathway (Bottom-Up): The LLM recommends nonlinear feature transformations (e.g., power-law terms, transcendental mappings) based on the problem context and the performance history of prior generations. These transformations are appended to the input matrix ( $X_{aug}$ ) and accumulate across generations. This shifts the search burden from constructing deep expression trees over raw variables to assembling shallow combinations over a rich, semantically informed feature set.

3. Search with PySR

The synthesized guidance (seeds, operator biases, and augmented features) steers a modified PySR engine. A complexity-biasing mechanism adjusts the structural penalty of expression trees: operators recommended by the LLM receive a negative bias (complexity discount), while physically implausible operators are penalized. A warm-start mechanism retains the best candidate from the previous generation as an elite member of the new population.

4. Multi-Dimensional Evaluation and Strategic Feedback

After each generation, the LLM acts as a post-generation critic. It evaluates Pareto-optimal candidates based on numerical accuracy, physical interpretability, and feature utility. These assessments are stored in a dynamic knowledge base, which informs the next iteration by refining seed expressions, adjusting operator biases, and prioritizing feature transformations. This creates a closed-loop, self-correcting refinement process.

Key Contributions

Semantic Seed Pathway: A mechanism leveraging domain-aware LLMs to propose dimensionally consistent functional skeletons. This prunes the search space of physically inconsistent expressions before numerical fitting begins and steers the search toward plausible operator combinations.
Structural Feature Pathway: A complementary pathway where the LLM recommends nonlinear feature transformations that accumulate over iterations. This reduces the structural depth required by the symbolic engine, shifting the search from deep tree construction to shallow assembly over an enriched basis.
Closed-Loop Iterative Refinement: A learning-from-results mechanism where the LLM evaluates candidates across multiple dimensions and updates a knowledge base. This transforms symbolic discovery from open-ended generation into an iterative, self-correcting process that converges toward optimal symbolic forms.

Experimental Results

InsightSR was evaluated on three benchmarks:

Feynman Benchmark (100 physics equations): Achieved a 95% exact recovery rate and an average $R^2$ of 0.9999, substantially outperforming state-of-the-art GP and neural-symbolic methods. It demonstrated robustness to moderate noise, maintaining high accuracy even at noise levels of 0.001.
LLM-SRBench (Cross-domain tasks): Achieved an 80.18% accuracy ( $Acc_{0.1}$ ) on the LSR-Transform task, a significant improvement over the previous best (LaSR at 50.45%). The method showed consistent performance across Chemistry, Biology, Physics, and Material Science domains.
Real-World Datasets: On datasets including Oscillator 1/2, E. coli, and Stress-Strain, InsightSR achieved the lowest Normalized Mean Squared Error (NMSE) in most test cases, often improving upon previous bests by several orders of magnitude. Crucially, it maintained strong out-of-distribution (OOD) generalization, avoiding the overfitting common in purely neural methods.

Significance and Claims

The paper claims that InsightSR offers a practical and effective path toward automated scientific discovery by strategically integrating LLM reasoning with established evolutionary search engines. Its significance lies in:

Synergistic Guidance: Successfully combining top-down semantic constraints with bottom-up feature engineering to address both the combinatorial explosion of the search space and the difficulty of maintaining physical consistency.
Efficiency and Parsimony: By shifting the search from deep trees over raw variables to shallow combinations over enriched features, the framework significantly reduces the structural complexity the engine must resolve.
Self-Correcting Discovery: The closed-loop feedback mechanism allows the system to learn from both successful patterns and failure modes, progressively narrowing the search toward physically meaningful expressions without relying on hard-coded constraints that limit adaptability.

The authors note that while the framework enhances search efficiency, its reliance on initial metadata suggests a potential sensitivity to problem descriptions, particularly when variable semantics are ambiguous. Future work may explore multi-modal priors to refine initialization and enhance robustness in the absence of explicit textual metadata.