In Context Learning and Reasoning for Symbolic Regression with Large Language Models

1. Problem Statement

Symbolic Regression (SR) is a machine learning technique aimed at discovering mathematical expressions that explain datasets by balancing accuracy and simplicity (parsimony). Unlike traditional regression, which fits constants to a pre-defined model structure, SR searches the "space of possible equations" to infer both the functional form and the parameters.

While existing SR methods (e.g., Genetic Algorithms, Bayesian methods, and pre-trained transformer models) are effective, they often lack reasoning capabilities and struggle to integrate scientific context (domain knowledge) without complex, bespoke software modifications. Furthermore, standard Large Language Models (LLMs) have been shown to be unreliable for SR because they tend to "hallucinate" numerical coefficients when asked to fit data directly, rather than generating the correct functional form.

The core problem addressed is: How can LLMs be effectively leveraged to perform symbolic regression by combining their natural language reasoning capabilities with rigorous numerical optimization, while incorporating scientific context to guide the search?

2. Methodology

The authors propose a hybrid workflow that decouples symbolic structure generation (handled by the LLM) from numerical parameter optimization (handled by external Python tools).

A. System Architecture

1. LLM Role: The LLM (GPT-4 or GPT-4o) is tasked with proposing mathematical expressions based on data and context. It does not optimize coefficients.
2. External Optimizer: A Python class using SciPy (Nelder-Mead with basin-hopping) takes the LLM's expressions, optimizes the numerical constants, and calculates the Mean Squared Error (MSE) and complexity.
3. Feedback Loop: The results (MSE, complexity, and optimized parameters) are formatted into a JSON dictionary and fed back to the LLM as part of an iterative prompt. The LLM is then instructed to propose improved expressions that optimize for both lower loss and lower complexity.

B. Prompt Engineering Strategies

To overcome LLM limitations, the authors implemented several key prompting techniques:

• Scratchpad (Chain-of-Thought): The LLM is instructed to first perform "data analysis" and "observations" in a scratchpad section before generating the final equation. This forces the model to reason about data patterns (e.g., inverse relationships, saturation) before committing to a formula.
• Context Integration: Natural language descriptions of the scientific domain (e.g., "nitrogen adsorption on mica," "pressure vs. loading") are provided alongside the data to constrain the search space toward physically meaningful equations.
• Bias Mitigation:
  • Two-Prompt Setup: An "Initial Prompt" generates unbiased expressions in LaTeX, while an "Iteration Prompt" receives previous results as Python scripts to avoid the model simply copying example syntax.
  • Format Adjustment: Switching from LaTeX to Python strings for GPT-4o reduced syntax errors.
• Constraint Enforcement: Scientific constraints (e.g., "loading must be zero at zero pressure," "monotonic non-decrease") are encoded as natural language instructions in the prompt rather than hard-coded mathematical constraints.

C. Complexity Definition

Complexity is defined as the number of nodes in the expression tree (operators, variables, and constants), weighted to penalize unary functions and binary operators.

3. Key Contributions

1. Hybrid SR Workflow: Demonstrated a robust pipeline where LLMs handle the creative/structural search and external tools handle numerical precision, preventing coefficient hallucination.
2. Context-Aware Reasoning: Showed that providing scientific context in natural language significantly improves the LLM's ability to rediscover known laws and generate physically consistent equations.
3. Scratchpad Efficacy: Validated that forcing LLMs to "think" (via scratchpad) about data patterns before generating equations leads to higher-quality functional forms.
4. Natural Language Constraints: Proved that complex mathematical constraints (like boundary conditions and monotonicity) can be effectively communicated via natural language prompts, simplifying the integration of domain knowledge compared to traditional SR software.
5. Evaluation on Real-World Data: Tested the approach on experimental datasets (Langmuir, Dual-site Langmuir, Nikuradse) rather than just synthetic benchmarks, including a dataset (Nikuradse) with no known target equation.

4. Results

The study evaluated GPT-4 and GPT-4o models on three datasets:

• Langmuir Adsorption Model:
  • The models successfully rediscovered the target equation ($q = c_1 p / (c_2 + p)$) in almost all cases.
  • Ablation Studies: Models performed best when all tools (Data + Context + Scratchpad) were enabled. Without context, models relied on general knowledge but often guessed incorrect adsorption models (e.g., Freundlich, BET). Without data, they relied heavily on context but lacked pattern recognition.
• Dual-Site Langmuir Model:
  • This is a harder problem where the target model is complex but fits the data similarly to simpler, physically inconsistent models.
  • GPT-4o outperformed GPT-4, successfully identifying the dual-site structure by analyzing saturation patterns in the scratchpad.
  • Constraint Impact: When natural language constraints (e.g., "passes through origin") were applied, the models generated significantly fewer unphysical expressions compared to unconstrained runs and traditional SR algorithms (PySR, BMS).
• Nikuradse Dataset (Flow in Rough Pipes):
  • A multivariate dataset with no known target equation.
  • The LLMs generated expressions with Mean Absolute Errors (MAE) of ~0.009–0.010.
  • While the BMS (Bayesian Symbolic Regression) algorithm found a slightly more accurate expression (MAE 0.0039), it required ~18,000 Monte Carlo steps. The LLM approach found competitive solutions from a pool of only ~153 expressions.
  • GPT-4o demonstrated better reasoning, generating accurate expressions at lower complexity than GPT-4.

5. Significance and Limitations

Significance:

• Accessibility: The natural language interface lowers the barrier to entry for symbolic regression, allowing domain scientists to incorporate theoretical constraints without writing complex code.
• Interpretability: The workflow produces human-readable equations that align with scientific theory, unlike "black box" neural networks.
• Efficiency: For complex problems, the LLM-guided search can find high-quality solutions with fewer evaluations than brute-force evolutionary algorithms, though it is currently more expensive per token than simple genetic algorithms.

Limitations:

• Test Set Leakage: The models may have "memorized" famous laws (e.g., Kepler's Law) from their training data, leading to false positives in rediscovery tasks. The authors note this was less of an issue for the adsorption and Nikuradse datasets.
• Cost and Scalability: Iterative prompting is computationally expensive and token-heavy, especially for large datasets.
• Control: LLMs struggle to strictly adhere to specific constraints on expression length or complexity via natural language prompts compared to deterministic algorithms.
• Performance Ceiling: While effective for simpler or medium-complexity problems, the approach does not yet outperform state-of-the-art SR programs (like BMS) on highly complex, noisy datasets where the target equation is extremely intricate.

Conclusion

The paper establishes that Large Language Models, when combined with external optimization tools and guided by chain-of-thought reasoning and scientific context, can serve as powerful engines for symbolic regression. They offer a unique advantage in integrating domain knowledge and generating interpretable, physically consistent equations, bridging the gap between data-driven discovery and theoretical science.