💻 computer science

Smooth multi-function evolutionary algorithm

This paper proposes a novel evolutionary algorithm for symbolic regression that utilizes a universal function template with smooth, single-variable parametrized translations between function pairs to reduce the function set and ensure derivative continuity, demonstrating its effectiveness on standard benchmarks.

Original authors: Peter Michalicek, Tomas Brandejsky

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Peter Michalicek, Tomas Brandejsky

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer science, there is a persistent challenge known as symbolic regression. Imagine a scientist who has collected a vast amount of data points—perhaps measurements of wind speed, pollen counts, or faculty salaries—and wants to find a single mathematical sentence that explains how these numbers relate to one another. The goal is to discover the hidden rule, the equation, that turns the input data into the correct output. To do this, computers often use a method inspired by natural selection, where they generate thousands of random mathematical sentences, test how well they fit the data, and then mix and mutate the best ones to create even better versions. However, this process is often clumsy. The computer treats the building blocks of these sentences—like addition, subtraction, or multiplication—as rigid, separate tools. Switching from one tool to another is like a sudden jump in the landscape of possibilities, making it difficult for the computer to find the smooth path to the perfect answer. It is a bit like trying to climb a mountain where the terrain suddenly shifts from a gentle slope to a vertical cliff; the climber struggles to find the right direction.

Researchers Peter Michalicek and Tomas Brandejsky from the University of Pardubice in the Czech Republic have proposed a new way to smooth out this jagged landscape. They developed a system where the computer does not have to choose between different mathematical tools as if they were distinct, separate options. Instead, they created a single, flexible tool that can morph smoothly from one operation to another. In their new approach, the computer uses a special control variable, a single dial that can be turned to gradually shift the behavior of the function. If the dial is set to one position, the tool acts like addition; if turned to another, it acts like multiplication; and in the middle, it performs a blend of both. This allows the computer to slide continuously between different mathematical behaviors rather than making abrupt, jarring jumps. By doing this, the researchers turned a difficult search for a complex tree-like structure into a simpler task of tuning a set of numbers, making the evolutionary process more efficient and less likely to get stuck in dead ends.

The researchers tested this idea by building a system they call a "smooth multi-function evolutionary algorithm." Instead of letting the computer randomly assemble a tree of different mathematical operations, they forced the computer to use a perfect, symmetrical tree where every internal node is this same flexible, shape-shifting tool. The only things that change during the learning process are the values of the control dials at each node and the specific numbers or variables at the bottom of the tree. This design simplifies the problem significantly. Because the structure of the tree is fixed and the transitions between operations are smooth, the computer can use standard optimization techniques to find the best settings, rather than relying on the more chaotic and unpredictable methods usually required for this type of problem. The researchers found that this approach allowed them to represent complex relationships using a single control variable for any number of basis functions, reducing the complexity of the search space.

To see if this method actually worked, the team ran their algorithm on a variety of standard datasets used to test symbolic regression, including data on wind patterns, pollen counts, and university faculty salaries. They compared their results against several other well-known methods, including systems that use traditional genetic programming and those that combine evolution with other optimization techniques. The results showed that while their method sometimes took longer to run on a standard computer processor, it was highly effective at finding accurate models. In some cases, particularly when given enough time to run through many generations of testing, their algorithm found solutions that were as good as, or better than, the established methods. For instance, on one dataset involving faculty salaries, their method achieved a high level of accuracy that matched the top performers. On another dataset regarding wind data, it produced results comparable to the best existing tools.

One of the most significant findings was related to how well the models generalized to new, unseen data. In machine learning, a common problem is overfitting, where a model memorizes the training data so perfectly that it fails to make accurate predictions on new information. The researchers measured this by comparing the error rate on the training data against the error rate on test data. Their smooth multi-function approach showed a very small increase in error when moving from training to testing, suggesting that the models it created were stable and robust. This indicates that the smooth transitions between mathematical operations helped the algorithm find solutions that captured the true underlying patterns of the data, rather than just fitting the noise. The researchers noted that their current implementation was written in a way that was not fully optimized for speed, which limited how many tests they could run in a fixed amount of time. They suggested that if the code were rewritten to run faster, perhaps by using specialized computer hardware, the performance could improve even further.

The study concludes that by replacing the rigid, jump-based transitions of traditional evolutionary algorithms with smooth, continuous ones, it is possible to make the search for mathematical models more efficient and reliable. The researchers demonstrated that a single control variable could effectively manage the transition between multiple different mathematical functions, reducing the complexity of the problem without sacrificing accuracy. While the method is not a magic bullet that solves every problem instantly, it offers a promising alternative for finding the hidden rules in data. The work suggests that the way we represent the building blocks of our models matters just as much as the algorithms we use to search for them. By smoothing out the landscape, the researchers have provided a clearer path for computers to follow, potentially leading to more accurate and reliable models in fields ranging from environmental science to economics. The code and results from this study are available for others to examine and build upon, inviting further exploration into this smoother way of evolving mathematical solutions.

Technical Summary: Smooth Multi-function Evolutionary Algorithm

Problem Statement
In Genetic Programming Algorithms (GPAs) applied to Symbolic Regression (SR), the search space consists of algebraic relations. Standard evolutionary operators (mutation and crossover) typically induce nonlinear "jump changes" in the fitness landscape when substituting functions or altering tree structures. This discontinuity makes it difficult to predict results, select candidates effectively, or estimate the impact of specific mutations, thereby impairing algorithmic efficiency. Furthermore, existing approaches to creating "universal" or "smooth" functions often require multiple control variables to transition between basis functions (e.g., one variable per function pair or triplet), leading to high-dimensional control spaces that complicate optimization. The paper addresses the need for a representation that reduces the dimensionality of the control space while maintaining smooth transitions between multiple basis functions.

Methodology
The authors propose a "Smooth Multi-function" (SM) representation that reduces the set of functions in a GPA to a single universal function controlled by a single control variable ( $\gamma$ ), regardless of the number of basis functions ( $n$ ) involved.

Smooth Multi-function Definition:
- Instead of using separate control variables for every function pair, the authors define a cyclic domain for a single control variable $\gamma$ .
- The domain of $\gamma$ is divided into $n$ zones, where each zone corresponds to a transition between neighboring function pairs (e.g., $f_1 \to f_2$ , $f_2 \to f_3$ , ..., $f_n \to f_1$ ).
- The transition is modeled using fuzzy logic concepts (fuzzification and defuzzification). The control variable $\gamma$ determines the weight of the neighboring functions in a linear interpolation.
- To ensure mathematical smoothness (specifically $C^1$ continuity where left and right derivatives are identical), the authors replace the standard triangular membership functions (which have discontinuous derivatives at peaks) with a Fourier series approximation. This allows the multifunction to be differentiable at all points, facilitating smoother optimization.
Evolutionary Strategy:
- The solution structure is represented as a perfect binary tree of a fixed depth.
- Crucially, all internal nodes (except the leaves) utilize the same single-type smooth multifunction.
- Because the structure is fixed and the nodes are uniform, the evolutionary process does not need to evolve the tree topology. Instead, it optimizes a flat vector containing:
  - The control variable values ( $\gamma$ ) for each multifunction node.
  - The terminal nodes (constants or variable indices).
- This transforms the problem from a structural search (typical in GP) into a parameter optimization problem.
- The authors utilize an Evolutionary Strategy (ES) with elitist selection. The crossover operator generates new parameter values by sampling from intervals defined by parent values (extended by an expansion rate), rather than simply swapping genetic material. Mutation is optional and applied to vector values.
Experimental Setup:
- The algorithm was tested on benchmarks from the SRBench suite.
- Variants were evaluated: SM with mutation (SM3, SM4), SM without mutation (SM3-NM, SM4-NM), and different tree depths.
- Comparisons were made against established SR methods: AFP, Bingo, EPLEX, PySR, and TIR.

Key Contributions

Dimensionality Reduction: The paper introduces a method to control a smooth transition between $n$ basis functions using only one control variable, significantly reducing the dimensionality of the fitness landscape compared to previous methods requiring $n-1$ or more variables.
Smoothness via Fourier Approximation: The authors provide a specific implementation of the multifunction that ensures $C^1$ continuity (smooth derivatives) by approximating the triangular membership function with a Fourier series, addressing the discontinuity issues of standard fuzzy transitions.
Structural Simplification: By fixing the tree structure to a perfect binary tree of uniform multifunctions, the authors shift the evolutionary burden entirely to parameter optimization. This eliminates the need for complex tree-based crossover and allows for efficient implementation on GPUs by representing individuals as flat vectors.

Results

Performance vs. Runtime: On a fixed runtime budget, the proposed SM algorithms were generally slower per generation than competitors (due to unoptimized Python implementation) and showed mixed results on larger datasets (e.g., 564 fried). However, when evaluated on a fixed number of generations (10,000), the SM algorithms became competitive. Specifically, SM3 achieved the lowest test Mean Squared Error (MSE) on the 230 machine cpu dataset after 10,000 generations.
Generalization: A significant finding was the generalization capability. The SM3 variant achieved the lowest ratio of Test MSE to Training MSE (1.138) among all evaluated methods. This indicates that the smooth multi-function representation produces models with a low tendency to overfit and good transferability to unseen data.
Mutation Impact: The effect of the mutation operator was inconsistent; in some datasets, mutation-free variants (e.g., SM4-NM) outperformed those with mutation, while the reverse was true for others.

Significance and Claims
The paper claims that this approach offers a convenient alternative to standard Genetic Programming for symbolic regression by:

Reducing Complexity: Replacing the structural evolution of trees with the optimization of a parameter vector in a smooth, continuous space ( $R^n$ ).
Enabling Hardware Acceleration: The vector-based representation eliminates the overhead of transferring tree structures between CPU and GPU, making it highly suitable for GPGPU implementation.
Improving Stability: The smoothness of the fitness landscape and the fixed structure appear to mitigate overfitting, leading to models that generalize well, even if the raw predictive accuracy on some large datasets is not yet superior to state-of-the-art methods like PySR.

The authors note that the current performance limitations are largely attributed to the unoptimized Python implementation and suggest that code optimization, parallelization, or GPU acceleration could substantially improve results under fixed runtime constraints.

Technical Summary: Smooth Multi-function Evolutionary Algorithm

More like this