💻 computer science

Symbolic Regression Using Synergistic Basis Functions

This paper introduces Synergistic Basis Functions Regression (SBFR), a non-evolutionary symbolic regression method that dynamically generates and selects synergistic basis functions to achieve competitive predictive performance with compact feature representations, significantly outperforming baseline models in accuracy while eliminating the need for computationally expensive evolutionary search.

Original authors: Hamide Sarvary Karizaky, Mohammad Mehdi Ebadzadeh

Published 2026-10-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamide Sarvary Karizaky, Mohammad Mehdi Ebadzadeh

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, researchers are constantly trying to find the hidden rules that connect one set of numbers to another. Imagine you have a list of inputs, like the ingredients in a cake, and a list of outcomes, like the height the cake rises. The goal is to write down the exact mathematical recipe that turns the ingredients into the result. This is called regression. For simple problems, a straight line might do the trick, but the real world is rarely that simple. Often, the relationship is twisted and tangled, requiring complex formulas to describe it accurately.

For decades, scientists have used a method called symbolic regression to solve this. Instead of just guessing a formula, these computer programs search through a vast library of possible mathematical expressions to find the one that fits the data best. The big advantage of this approach is that the final answer is a clear, written equation that a human can read and understand. However, finding that perfect equation is like looking for a specific needle in a haystack that keeps growing. The traditional way to do this involves a process similar to evolution, where thousands of candidate formulas compete, mutate, and reproduce over many generations. While this can work, it is incredibly slow and computationally expensive, often requiring massive amounts of computer power just to find a solution that might be unnecessarily complicated.

A team of researchers at Amirkabir University of Technology in Iran has proposed a different way to tackle this problem. They introduced a new method called Synergistic Basis Functions Regression, or SBFR, which skips the slow, evolutionary search entirely. Instead of letting thousands of formulas fight it out, this new approach builds the solution piece by piece, like a craftsman carefully selecting and fitting individual tools. The core idea is to generate candidate mathematical pieces, called basis functions, and then test how well they work together. The researchers realized that a single mathematical piece might not seem very useful on its own, but when combined with the right partner, it could become incredibly powerful. This concept is known as synergy: the whole is greater than the sum of its parts.

The researchers designed their system to act like a dynamic editor. It starts with an empty set of tools and begins generating new mathematical expressions using a standard method that creates diverse structures. For every new expression it creates, the system asks two questions. First, does this new piece improve the model if we simply add it to our current collection? Second, could this new piece do a better job if we swapped it out for one of the pieces we already have? The system measures the improvement in prediction accuracy for both scenarios. If adding the new piece helps, it is kept. If swapping it with an existing piece makes the model even better, the old piece is discarded and replaced. This process repeats, constantly refining the collection of mathematical tools until the model is as accurate and compact as possible.

To test if this new method worked, the researchers applied it to nine different datasets, ranging from synthetic problems designed to test mathematical skills to real-world data involving things like concrete strength and property prices. They compared their new system against thirteen other methods, including the traditional evolutionary approaches and standard machine learning models. The results showed that the new method was highly effective. On average, the models built with this new approach reduced their prediction errors by about forty percent compared to the standard methods. They also improved the ability of the models to explain the data by over twenty percent. Crucially, the new method achieved this without needing the heavy computational cost of evolutionary search.

Perhaps the most striking finding was how efficient the new system was at keeping the models simple. While some of the traditional methods produced models with hundreds or even thousands of mathematical terms, making them difficult for humans to interpret, the new method typically selected only between four and twelve pieces to build a working model. This resulted in a final equation that was not only accurate but also short enough to be easily understood and inspected. The researchers found that the best balance of accuracy and simplicity was achieved when the new method was paired with a specific type of machine learning algorithm known as a Random Forest, which acted as the final decision-maker using the selected mathematical pieces.

The study also highlighted that the success of the method depended on how well the mathematical pieces worked together. A piece that seemed weak in isolation could become a star player when paired with the right neighbors. By focusing on these cooperative relationships rather than just the strength of individual pieces, the system avoided the clutter of redundant or useless formulas. This approach proved to be a viable alternative to the traditional, brute-force evolutionary methods, offering a way to discover clear, mathematical rules from complex data without the massive computational overhead. The work suggests that by thinking about how data features interact and support one another, we can build smarter, cleaner, and more understandable models for the future.

Technical Summary: Symbolic Regression Using Synergistic Basis Functions

Problem Statement

Symbolic Regression (SR) aims to discover explicit mathematical relationships between input variables and a continuous target without assuming a predefined functional form. While SR offers superior interpretability compared to "black-box" models like Artificial Neural Networks (ANNs), it faces significant challenges due to its vast and complex search space. Traditional evolutionary approaches, particularly Genetic Programming (GP), often suffer from high computational costs, slow convergence, and the tendency to produce unnecessarily complex representations (bloat). Furthermore, existing non-evolutionary methods often struggle with basis function selection, frequently evaluating candidate functions based on individual predictive contribution rather than their cooperative or synergistic effects when combined with an existing set of features. This oversight can lead to redundant representations and missed opportunities for discovering valuable function combinations.

Methodology: Synergistic Basis Functions Regression (SBFR)

The paper proposes Synergistic Basis Functions Regression (SBFR), a non-evolutionary framework designed to construct compact and interpretable symbolic models. Unlike GP, which evolves populations of complete expressions, SBFR iteratively generates and selects individual basis functions based on their contribution to a regression model in the context of already selected features.

Core Mechanisms

Candidate Generation:
SBFR generates candidate symbolic basis functions (represented as tree structures) using the Ramped Half-and-Half (RHH) procedure. This method combines "full" and "grow" strategies to produce expressions with diverse structures and depths from a predefined set of functions (e.g., $+$ , $-$, $\times$ , $\div$ ) and terminals (input variables and random constants).
Synergy-Based Selection:
Instead of selecting functions solely based on standalone performance, SBFR evaluates candidates using two complementary metrics relative to the current set of selected basis functions ( $FV$ ):
- Synergy by Adding ( $S_{Add}$ ): Measures the relative reduction in Root Mean Squared Error (RMSE) when a candidate is added to the current feature set.
- Synergy by Replacing ( $S_{Rep}$ ): Measures the relative reduction in RMSE when a candidate replaces an existing basis function in the set.
A candidate is accepted only if the improvement exceeds a predefined threshold (improvethreshold). The algorithm dynamically updates the feature set by either adding a new function or replacing an existing one if the replacement yields a greater synergistic improvement.
Model-Agnostic Framework:
SBFR is designed to be model-agnostic. The constructed symbolic basis functions serve as transformed features for a downstream regression model. In the experiments, the authors evaluated SBFR using various downstream learners, including Linear Regression (Ridge), Support Vector Regression (SVR), K-Nearest Neighbors (KNN), Decision Trees (DT), Random Forest (RF), and Multi-Layer Perceptrons (MLP).
Stopping Criteria:
The iterative process terminates when a maximum number of attempts is reached, no significant improvement is observed over a consecutive number of evaluations, or the target error (RMSE = 0) is achieved.

Key Contributions

The authors identify three primary contributions:

Extension of Synergy to Regression: The work extends the concept of synergistic feature construction, previously applied to classification, to the domain of Symbolic Regression, resulting in the non-evolutionary SBFR framework.
Dynamic Selection Strategy: The introduction of a dynamic selection mechanism that retains complementary basis functions and reduces redundancy by explicitly evaluating the cooperative contribution of candidates, avoiding the need for population-based evolutionary search.
Comprehensive Evaluation: A rigorous evaluation of SBFR across nine benchmark regression datasets (both synthetic and real-world) against 13 baseline methods, assessing both predictive performance and model compactness.

Experimental Results

The study evaluated SBFR on nine datasets: Concrete, Pollen, RatPol2D, Real Estate, Toxicity, X05Y1Z15, bm1, bm4, and keijzer5. The results were compared against conventional regressors and state-of-the-art SR methods (including GP variants like GSGP, M3GP, EvoF, and non-evolutionary methods like FFX).

Predictive Performance

Error Reduction: Across the nine datasets, SBFR-based configurations achieved average relative reductions of 40.14% in RMSE and 41.53% in Mean Absolute Error (MAE) compared to their corresponding baseline regressors.
$R^2$ Improvement: There was an average relative improvement of 21.56% in the coefficient of determination ( $R^2$ ).
Statistical Significance: The SBFR-RF (Random Forest downstream) configuration achieved the second-best average rank for RMSE (3.78), statistically comparable to the top-performing evolutionary method EvoF (rank 2.89), but significantly outperforming the standalone Random Forest (rank 5.22).
Metric Consistency: SBFR-RF maintained strong performance across MAE and $R^2$ metrics as well, consistently outperforming its baseline and competing with top evolutionary methods.

Model Compactness and Interpretability

Feature Dimensionality: A critical advantage of SBFR is the compactness of the resulting feature space. While evolutionary methods like GSGP and EvoF often produced hundreds of features (e.g., GSGP median >600 features), SBFR variants typically retained only 4–12 basis functions (depending on the downstream model and dataset).
Compactness Rankings: SBFR-SVR achieved the best average rank (2.44) for feature-space compactness, followed closely by SBFR-RF (2.56).
Trade-off: The paper notes a trade-off where SBFR-SVR offers the most compact representations but slightly less competitive predictive performance than SBFR-RF. SBFR-RF is highlighted as offering the most favorable balance between predictive accuracy and feature compactness.

Example Analysis

An illustrative example on the RatPol2D dataset showed SBFR-RF constructing eight symbolic basis functions. These included simple linear transformations (e.g., $3x_0$ ) and nonlinear interactions (e.g., $x_0(x_1 - x_0)$ ), demonstrating the ability to create explicit, interpretable mathematical expressions that capture complex relationships.

Significance and Claims

The paper claims that SBFR addresses the limitations of evolutionary SR by replacing computationally expensive population searches with a synergy-guided, iterative basis function selection process. The significance of the work lies in:

Efficiency: It avoids the high computational cost of evaluating large populations of candidate expressions.
Interpretability: It produces compact symbolic representations that are easier to interpret than the massive feature sets often generated by evolutionary methods.
Effectiveness: It demonstrates that explicitly evaluating the synergistic contribution of functions leads to better predictive performance and more compact models than methods relying solely on individual feature selection or standard evolutionary search.

The authors conclude that SBFR provides a viable, non-evolutionary alternative for Symbolic Regression that successfully balances predictive performance, model compactness, and interpretability, particularly when paired with ensemble learners like Random Forest.