🤖 machine learning

When Good Equations Get Bad Scores: Improving Symbolic Regression Through Better Parameter Optimization

This paper introduces SAGE-Fit, a plug-and-play framework that leverages structural and semantic priors of symbolic expressions to overcome the "Good Structure, Bad Score" bottleneck in Symbolic Regression, thereby significantly improving parameter optimization and overall search performance.

Original authors: Boxiao Wang, Kai Li, Zhiwei Chen, Yang Huang, Runxiang Wang, Ziwen Zhang, Yifan Zhang, Jian Cheng

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Boxiao Wang, Kai Li, Zhiwei Chen, Yang Huang, Runxiang Wang, Ziwen Zhang, Yifan Zhang, Jian Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a pile of clues (data) and you need to find the secret rule (an equation) that explains how those clues fit together. This is what Symbolic Regression does: it tries to discover the hidden mathematical laws of nature from data.

However, the paper argues that most detectives are using a flawed strategy. They are great at guessing the shape of the rule, but terrible at checking if that shape actually works.

Here is the breakdown of the problem and the paper's solution, using simple analogies.

The Problem: "Good Shape, Bad Score"

Think of the detective's job as a two-step process:

The Outer Loop (The Architect): This part guesses the structure of the equation. It's like sketching a blueprint for a house. "Maybe the roof is a triangle? Maybe the walls are made of brick?"
The Inner Loop (The Builder): Once the blueprint is drawn, this part tries to fit the actual numbers (parameters) into it to see if the house stands up. It's like trying to hammer in the nails and mix the concrete to match the sketch.

The Flaw:
The paper says the "Architect" has become very smart, but the "Builder" is still using a cheap, fast, and clumsy tool (standard math solvers like BFGS).

Because the math involved is incredibly complex (like a landscape full of deep valleys and sharp spikes), the clumsy builder often gets stuck in a small hole. They look at a perfect blueprint (a correct equation structure) and say, "This is a terrible house!" simply because they couldn't finish the construction properly.

The paper calls this the "Good Structure, Bad Score" problem. The detective discards the correct answer because the builder failed to prove it works, leading the search down the wrong path.

The Solution: SAGE-Fit

The authors propose a new, super-smart builder called SAGE-Fit. Instead of treating the equation like a black box, SAGE-Fit understands the "language" of the equation. It uses three special tricks to fix the building process:

1. The "Separation" Trick (Structure-Aware)

Imagine you are assembling a complex toy. Some parts are easy to snap in (linear), while others require precise twisting (non-linear).

Old Way: The builder tries to twist and turn every part at once, getting confused and stuck.
SAGE-Fit: It looks at the blueprint, identifies the easy parts, and snaps them in perfectly using a simple, automatic tool. This leaves only the tricky, twisting parts for the builder to focus on. This makes the job much smaller and easier.

2. The "Map" Trick (Semantics-Guided)

Imagine you are trying to find the best spot to set up a campfire in a forest.

Old Way: The builder picks random spots based on coordinates (e.g., "5 steps North, 3 steps East"). But in this forest, "5 steps North" might be the same spot as "5 steps North + 360 degrees" (a circle). The builder wastes time checking the same spot over and over.
SAGE-Fit: Instead of looking at coordinates, it looks at the actual fire. It asks, "Does this spot produce a warm, bright flame?" It picks starting spots that produce different types of fires. This ensures the builder explores truly different areas of the forest, avoiding the trap of checking the same spot twice.

3. The "Rolling Ball" Trick (Projected Gauss-Newton)

Once the builder has a few good starting spots, they need to find the absolute best spot (the bottom of the valley).

Old Way: The builder rolls a ball down the hill, but the hill is so bumpy that the ball gets stuck in tiny potholes.
SAGE-Fit: It uses a special, curved ramp that knows exactly how the hill is shaped. It guides the ball smoothly and quickly to the very bottom of the deepest valley, ensuring it finds the best possible fit every time.

The Results

The authors tested this new builder (SAGE-Fit) by plugging it into several existing detective teams (different AI systems).

The Outcome: When the teams used SAGE-Fit, they found the correct "secret rules" much more often.
The Proof: They discovered that in some cases, the old builders were throwing away nearly 50% to 65% of the correct answers just because they couldn't build them properly. SAGE-Fit saved these "good" answers, allowing the detectives to solve the mystery.

Summary

The paper doesn't invent a new way to guess the equation; it invents a much better way to check if the guess is right. By understanding the unique shape of mathematical equations, SAGE-Fit ensures that good ideas aren't discarded due to bad math, leading to faster and more accurate discovery of scientific laws.

Technical Summary: When Good Equations Get Bad Scores

Problem Statement

Symbolic Regression (SR) aims to discover interpretable mathematical equations from observational data. Most modern SR methods operate within a bi-level optimization framework: an outer loop searches for discrete equation structures (e.g., expression trees), while an inner loop optimizes the continuous parameters (constants, coefficients) for a given structure.

The paper identifies a critical bottleneck termed "Good Structure, Bad Score." In this phenomenon, structurally correct equations receive poor fitness scores because the inner-loop parameter optimization fails to locate the global minimum. This failure stems from:

Non-Convex Landscapes: Nested nonlinear operators (e.g., exponentials, trigonometric functions) create highly non-convex loss surfaces with numerous local minima, often described as a "Needle-in-a-Basin" topology.
Inadequate Solvers: Current SR systems rely on fast, generic local solvers (e.g., BFGS, Nelder-Mead) that are initialization-sensitive. These solvers frequently converge to suboptimal local minima or stall on flat plateaus, assigning unjustifiably low scores to correct structures.
Search Misguidance: The outer loop, misled by these poor scores, discards promising candidates, preventing convergence to the true physical law and wasting computational resources.

The authors argue that the bottleneck has shifted from generating valid structures to accurately evaluating them.

Methodology: SAGE-Fit

To address this, the authors propose SAGE-Fit (Structure-Aware and Semantics-Guided Evaluator for Symbolic Regression), a unified parameter fitting framework designed as a plug-and-play module. SAGE-Fit exploits two "native priors" inherent to symbolic expressions: algebraic structure and functional semantics.

The framework consists of three integrated modules:

1. Structure-Aware Dimensionality Reduction (Module 1)

Concept: Exploits the algebraic structural prior. Not all parameters in an equation are non-linear; some are conditionally linear (e.g., amplitudes in $a \sin(bx+c)$ ).
Mechanism: The system parses the Abstract Syntax Tree (AST) of the candidate equation to identify conditionally linear parameters. It employs Variable Projection (VarPro) to deterministically decouple these linear parameters ( $\alpha$ ) from the non-linear ones ( $\beta$ ).
Effect: For any fixed non-linear parameter set $\beta$ , the optimal linear parameters $\alpha^*$ are solved in closed form via linear least squares. This collapses the high-dimensional mixed search space into a lower-dimensional, purely non-linear manifold, eliminating spurious linear-to-non-linear coupling that causes ill-conditioning.

2. Semantics-Guided Initialization (Module 2)

Concept: Exploits the functional semantic prior. Distance in parameter space is deceptive (e.g., $\sin(x)$ and $\sin(x+2\pi)$ have different parameters but identical functions). Random parameter sampling leads to "basin collapse," where multiple starting points converge to the same local minimum.
Mechanism: The authors introduce Function-Space Farthest-Point Sampling (FS-FPS).
1. Generate an oversampled set of candidate parameter vectors.
2. Map these vectors to their predicted output vectors (functional fingerprints) on the training data.
3. Select a diverse set of starting points that are maximally separated in the function space (normalized Euclidean distance of outputs), rather than parameter space.
Effect: This ensures that multiple starting points explore genuinely distinct attractor basins, significantly reducing the risk of missing the global optimum due to poor initialization.

3. SR-Specialized Local Attraction (Module 3)

Concept: Even on the reduced manifold, the landscape remains non-convex. Standard gradient descent is inefficient.
Mechanism: SAGE-Fit employs a Projected Gauss–Newton method with Trust Region Reflective (TRF) damping. The Jacobian is computed implicitly, accounting for the dependence of the linear parameters on the non-linear ones.
Effect: This provides curvature-aware descent on the irreducible non-linear manifold, enabling rapid convergence to the global optimum from the high-quality seeds provided by Module 2.

Key Contributions

Diagnosis of the Bottleneck: The paper provides a landscape-level analysis characterizing SR optimization failures into "Needle-in-a-Basin" topologies for correct structures and diverse, unstable behaviors for partially correct structures.
SAGE-Fit Framework: A novel, plug-and-play evaluator that integrates structural priors (via VarPro) and semantic priors (via FS-FPS) to overcome the limitations of generic optimizers.
Empirical Validation: Extensive experiments across diverse SR paradigms (Genetic Programming, Reinforcement Learning, and LLM-based search) demonstrate that upgrading the inner-loop optimizer alone significantly improves performance without modifying the outer-loop search logic.

Experimental Results

The authors evaluated SAGE-Fit on the LSR-Synth dataset (128 equation discovery problems) across four domains: Chemistry, Biology, Physics, and Material Science. They tested it as a replacement for the inner-loop optimizer in four baseline systems: PySR, uDSR, LaSR, and LLM-SR.

Symbolic Accuracy (SA): The "Plus" variants (baselines + SAGE-Fit) consistently outperformed baselines. For example, PySR+ increased Symbolic Accuracy from 2.78% to 8.33% (a 3x improvement), and uDSR+ achieved the highest SA of 13.89% on the LSR-Synth dataset.
Numerical Fidelity: The "Plus" variants showed significant improvements in Accuracy to Tolerance ( $Acc_{0.1}$ ) and Normalized Mean Squared Error (NMSE), indicating that the search was guided toward structurally superior forms rather than just better-fitting incorrect ones.
Evaluator Fidelity Analysis:
- Lost Rate: In LLM-SR trajectories, the original evaluator caused a "lost rate" of up to 61.79% in Material Science, meaning promising candidates were discarded due to poor fitting. SAGE-Fit reduced this significantly.
- Candidate-Level Analysis: On the Structure-Progress Bank (a benchmark of 200 intermediate candidates), SAGE-Fit reduced the "missed progress" rate from 51.5% (using standard multi-start BFGS) to 8.0%.
- Log-Ratio Improvement: Replacing the original evaluator with SAGE-Fit resulted in median log-ratio improvements in NMSE of 1.91 (Biology), 1.59 (Chemistry), 0.69 (Physics), and 5.67 (Material Science), indicating error reductions of orders of magnitude in some domains.

Significance and Claims

The paper claims that the fidelity of the inner-loop evaluator is a major bottleneck in closed-loop symbolic regression, particularly for LLM-guided approaches. The significance of this work lies in demonstrating that:

Inner-loop optimization is a standalone lever: Enhancing parameter fitting alone can substantially strengthen mainstream SR systems without requiring complex changes to the symbolic search logic (e.g., new evolutionary operators or LLM prompts).
Priors matter: Treating symbolic expressions as black boxes ignores critical algebraic and semantic information. Explicitly leveraging these priors resolves the "Good Structure, Bad Score" phenomenon.
Universal Improvement: The approach is "plug-and-play," universally improving the performance of diverse SR systems (from evolutionary to LLM-based) by ensuring that structurally correct candidates are not prematurely discarded due to optimization failures.

The authors conclude that by mitigating the "Needle-in-a-Basin" pathology and preventing basin collapse, SAGE-Fit enables SR systems to realize their full potential in discovering complex scientific laws.