🤖 machine learning

Decomposable Neural Symbolic Regression

This paper introduces a decomposable neural symbolic regression method that leverages Multi-Set Transformers, genetic algorithms, and genetic programming to distill opaque regression models into accurate, interpretable multivariate expressions that consistently recover the original mathematical structure while maintaining competitive predictive performance.

Original authors: Giorgio Morales, John W. Sheppard

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Giorgio Morales, John W. Sheppard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of science, computers have become incredibly powerful tools for finding patterns in data. They can look at thousands of measurements from the natural world and build complex models that predict future outcomes with high accuracy. These models are often called "opaque" because, while they work well, their internal logic is so tangled and complicated that humans cannot easily understand how they reach their conclusions. They are like a black box: you put data in, and an answer comes out, but the path in between remains hidden. For scientists studying physics, biology, or engineering, this lack of transparency is a major hurdle. To truly understand the universe, researchers need more than just a prediction; they need the governing equations, the clear mathematical sentences that describe how nature works. This is the goal of symbolic regression, a field dedicated to finding simple, readable formulas that explain observed data, rather than just memorizing it.

A team of researchers at Montana State University has developed a new way to tackle this problem, offering a method that can peel back the layers of these complex computer models to reveal the simple mathematical truths underneath. Their approach, which they call SeTGAP, starts by training a standard, highly complex computer model to learn the behavior of a system. Once this "opaque" model is trained and working, the researchers do not try to reverse-engineer its millions of internal settings. Instead, they treat the model as a source of truth and ask it to explain itself, one variable at a time. They systematically isolate each input factor, such as temperature or pressure, and ask the model how the output changes when only that single factor moves. By doing this, the system generates a series of simple, single-variable sketches that describe the shape of the relationship for each piece of the puzzle.

The researchers then use a sophisticated process to stitch these individual sketches together. Imagine trying to understand a complex machine by first figuring out how each gear turns on its own, and then seeing how they lock together. The team uses a combination of artificial intelligence and evolutionary algorithms to merge these single-variable sketches into a complete, multi-variable formula. They do not simply throw all the pieces together at once, which often leads to confusion and overly complicated results. Instead, they add variables one by one, ensuring that the structure of the formula remains clean and logical at every step. This method allows them to preserve the original functional relationships they identified early on, preventing the final equation from becoming a messy tangle of numbers that fits the data but makes no sense.

When the team tested this method against a wide variety of problems, from synthetic mathematical challenges to real-world physics equations, the results were striking. On the synthetic problems, their method successfully recovered the correct mathematical structure of the underlying system in every single test case they ran. In contrast, other leading methods, including those that rely on deep learning or traditional evolutionary computing, frequently failed to find the right form, often producing expressions that were either too complex or simply wrong, even if they predicted the numbers correctly. The new approach proved particularly robust when the data contained noise or errors, a common occurrence in real-world measurements. While other methods struggled to generalize beyond the data they were trained on, the equations discovered by this method held up well when tested on new, unseen ranges of values.

The researchers also applied their technique to a famous collection of physics equations known as the Feynman dataset, which contains laws describing everything from gravity to the behavior of light. On this benchmark, their method achieved a high rate of success in identifying the correct equations, ranking second in symbolic solution recovery with approximately 61.2% successful recoveries, and demonstrating competitive predictive performance relative to many established techniques. The study suggests that by breaking a complex problem into smaller, manageable parts and then carefully reassembling them, it is possible to bypass the limitations of current artificial intelligence. This does not mean the computers are less powerful; rather, it means the method of asking them to explain themselves is more effective. The work demonstrates that it is possible to extract clear, human-readable laws of nature from the deep, hidden layers of modern machine learning, turning opaque predictions into transparent understanding.

Technical Summary: Decomposable Neural Symbolic Regression (SeTGAP)

Problem Definition

Symbolic Regression (SR) aims to discover interpretable mathematical expressions that capture the underlying relationships in observed data, serving as a transparent alternative to "opaque" deep learning models. However, existing SR methods face two primary challenges:

Structural Accuracy vs. Prediction Error: Most methods prioritize minimizing prediction error, often resulting in overly complex expressions that fit the data well but fail to recover the true governing equations.
Simultaneous Variable Processing: Current neural SR approaches and traditional Genetic Programming (GP) often process all variables simultaneously. This leads to a combinatorial explosion in the search space and frequently fails to correctly identify the functional form between individual variables and the system's response, particularly in non-separable systems.

Furthermore, while high-performing opaque models (e.g., neural networks) can effectively approximate complex systems, extracting interpretable governing equations from them remains difficult.

Methodology: SeTGAP

The authors propose SeTGAP (Symbolic Regression using Transformers, Genetic Algorithms, and genetic Programming), a decomposable, post-hoc explainable SR framework. Instead of searching for a single multivariate expression directly, SeTGAP distills a trained opaque model into a multivariate expression through a structured, incremental pipeline:

1. Univariate Skeleton Prediction (Multi-Set Transformer)

The method begins by isolating the influence of each independent variable $x_v$ on the opaque model's response.

Data Generation: For each variable, the method generates multiple sets of input-response pairs where $x_v$ varies while all other variables are fixed to random values. This creates a Multi-Set Symbolic Skeleton Prediction (MSSP) problem.
Model: A Multi-Set Transformer is employed to predict a univariate symbolic skeleton for each variable. This architecture is trained on synthetic data to identify the shared functional form across multiple input sets, effectively capturing how $x_v$ influences the response regardless of the specific values of other variables.
Candidate Generation: Using a diverse beam search, the transformer generates multiple candidate skeletons per variable. These are filtered to remove duplicates and mathematically equivalent forms (e.g., converting sine to cosine via phase shifts).

2. Incremental Merging via Genetic Programming

Once univariate skeletons are identified, they are merged into a multivariate expression using a cascade approach:

Recursive Merging (Algorithm 2): The method defines a recursive procedure to combine two skeletons ( $e_1$ and $e_2$ ) while strictly preserving their original univariate structures. Based on a structural decomposition proposition, the algorithm explores valid mathematical combinations (e.g., wrapping products, summing terms) where constants in one skeleton are replaced by sub-trees of the other.
Evolutionary Selection (Algorithm 3): A population of candidate merged skeletons is generated stochastically. A constrained evolutionary process (Genetic Programming) is applied where the structure of the skeletons is fixed, and only numerical coefficients are optimized via a Genetic Algorithm (GA). This decouples structural exploration from parameter optimization.
Cascade Strategy: Variables are merged sequentially, one by one, ranked by the quality of their initial univariate skeletons. This incremental approach avoids the combinatorial explosion of merging all variables at once and ensures that the contribution of each variable is evaluated in the context of previously merged variables.

3. Final Coefficient Optimization

The top-performing multivariate skeletons undergo a final global optimization pass on the original dataset to refine numerical coefficients, minimizing the Mean Squared Error (MSE).

Key Contributions

Decomposable Framework: SeTGAP shifts the paradigm from simultaneous multivariate search to a decomposable process, first identifying univariate functional forms and then merging them. This reduces the search space complexity and mitigates the risk of getting trapped in local minima.
Structure Preservation: Unlike standard GP or end-to-end neural SR, SeTGAP enforces structural constraints during the merging process. It ensures that the final multivariate expression remains aligned with the individual functional relationships identified by the Multi-Set Transformer.
Post-Hoc Explainability: The method is designed to distill opaque models, providing human-readable mathematical surrogates that explain the learned function of a pre-trained neural network.

Experimental Results

The authors evaluated SeTGAP on 13 synthetic non-linear problems and the Feynman dataset, comparing it against two GP-based methods (PySR, TaylorGP), three neural SR methods (NeSymReS, E2E, TPSR), and a hybrid approach (uDSR).

Functional Form Recovery: SeTGAP consistently recovered the correct functional form of the underlying equations across all 13 synthetic problems and all 10 experimental iterations (100% success rate). In contrast, competing methods failed to recover the correct structure in more than 7 out of 13 problems in any single iteration. Notably, some methods had applicability constraints; for instance, NeSymReS could not be applied to problems E4 and E5 due to its limitation to systems with at most three variables.
Extrapolation Performance: SeTGAP demonstrated lower or comparable extrapolation errors compared to benchmarks. The authors attribute this to the method's focus on structural accuracy rather than pure error minimization, which prevents overfitting to the training domain.
Feynman Dataset: On the Feynman benchmark, SeTGAP achieved a symbolic solution recovery rate of approximately 61.2%, ranking second among tested methods. While its aggregate $R^2$ and MSE rankings appeared lower in some aggregated metrics, statistical significance tests (Tukey's HSD and Kruskal-Wallis) indicated that SeTGAP achieved the lowest or statistically comparable performance in the majority of individual problems (73/78 for $R^2$ and 46/78 for MSE).
Noise Robustness: The method demonstrated robustness under varying noise levels, though performance degraded as noise increased, particularly for functions containing exponential terms where small coefficient errors lead to large deviations.
Limitations on Complexity: While highly effective on the synthetic and Feynman datasets, the method's performance is tied to the complexity of the skeletons it can process. The authors note that SeTGAP fails on specific complex benchmark problems, such as F4 of the SRBench++ benchmark, where the underlying function requires a number of operators (eight) and unary operator nesting that exceeds the training limits of the Multi-Set Transformer (trained on expressions with up to seven operators).

Significance and Claims

The paper positions SeTGAP not necessarily as a state-of-the-art method for raw predictive accuracy, but as a superior approach for interpretability and structural recovery.

Faithfulness to Governing Equations: The primary claim is that by prioritizing the identification of correct functional forms over mere error minimization, SeTGAP produces expressions that are faithful to the underlying physical or mathematical laws of the system.
Generalization: The authors argue that structural accuracy inherently leads to better generalization (extrapolation) capabilities, as the learned expressions capture the true relationships rather than data-specific noise.
Limitations: The authors modestly acknowledge limitations, including the computational cost of the multi-stage optimization process and the dependency on the complexity of the Multi-Set Transformer's training vocabulary. They note that the method is less efficient than direct SR approaches but argue that the trade-off is warranted for scientific discovery where reliable, interpretable equations are the goal.

In conclusion, SeTGAP offers a novel pathway to bridge the gap between high-performance opaque models and the need for interpretable, governing equations in scientific domains.