🤖 machine learning

Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

This paper demonstrates that fixed-depth symbolic regression can effectively discover compact, high-performing neural network weight-update rules that significantly outperform standard hand-designed optimizers on small-scale benchmarks, suggesting a promising lightweight approach for automated optimizer discovery.

Original authors: Charles Brum, Edward Finkelstein

Published 2026-07-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Charles Brum, Edward Finkelstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to paint a picture, but the robot is clumsy and keeps smudging the canvas. To fix this, you need a set of instructions—a recipe—that tells the robot exactly how to adjust its brushstrokes to get closer to the masterpiece. In the world of artificial intelligence, these instructions are called "weight-update rules." They are the mathematical steps a computer takes to learn from its mistakes. For decades, scientists have hand-crafted these recipes, mixing in ingredients like "momentum" (to keep the robot moving in a good direction) or "adaptive rates" (to slow down when things get tricky). But here's the big question: Are we missing a better recipe? What if there's a secret, super-efficient way to teach the robot that no human has thought to write down yet? This is the playground of "symbolic regression," a technique that acts like a digital alchemist, searching through endless combinations of math symbols to find the perfect formula for learning.

In this paper, two researchers decided to let a computer do the cooking. Instead of guessing which ingredients make the best optimizer, they used a method called symbolic regression to hunt for new weight-update rules. Think of it as a genetic algorithm that breeds millions of different mathematical expressions, testing each one to see if it helps a small neural network learn a task faster and more accurately than the best human-designed rules. They didn't just tweak the existing recipes; they let the computer invent entirely new ones using a toolbox of common math operations (like adding, multiplying, or taking square roots) and standard ingredients like gradients and momentum.

The results were surprisingly tasty. Out of 30 different learning challenges, the computer-discovered rules beat the best human-tuned optimizers in 25 cases. When the computer found a winner, it shaved off an average of 44.47% of the error (measured as Mean Squared Error) compared to the standard methods. However, the paper is careful to note that these experiments were run on small, simple networks and only for 10 epochs (a single training round). The authors suggest that while these new rules are compact and effective for these specific tasks, we don't know yet if they will hold up in the messy, complex real world of massive deep learning.

The discovered rules didn't all look the same. Some were simple, while others were wild mixes of trigonometric functions, exponentials, and rational expressions. Yet, many shared a common spirit: they combined momentum (like a rolling ball gaining speed) with adaptive normalization (adjusting the step size based on past performance). The researchers found that the computer could stumble upon these effective, interpretable formulas without being explicitly told to do so. It suggests that the space of "how to learn" is vast and full of hidden gems that human intuition might overlook. But before we throw away our old cookbooks, the authors warn that these findings are a promising start, not a final victory. The rules need to be tested on bigger networks and longer training sessions to see if they are truly the future of AI learning or just a clever trick for small puzzles.

Technical Summary: Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

Problem Statement

Training feed-forward neural networks relies on weight-update rules (optimizers) to minimize error functions. While standard optimizers like SGD, Momentum, Adam, and AdaGrad are widely used, they are often heuristically designed. The authors investigate whether symbolic regression can algorithmically discover explicit, compact weight-update rules that empirically outperform these hand-designed, hyperparameter-tuned optimizers on small-scale symbolic regression benchmarks, without significantly increasing computational complexity.

Methodology

Search Space and Representation

The study utilizes Symbolic Regression to search for weight-update rules represented as fixed-depth expression trees.

Operators: The search space includes a fixed set of unary operators (e.g., ln, exp, sin, tanh, sqrt), binary operators (+, -, *, /, ^), and specific hyperparameters (η, θ, ϵ, γ, β1, β2).
Operands: The leaf nodes of the expression trees are derived from quantities common in existing optimizers. These include current weights, gradients, velocity terms, moving averages of gradients (first and second moments), and accumulated squared gradients. The operands are defined to reflect the mechanics of Heavy-Ball, Nesterov, AdaGrad, RMSProp, Adadelta, Adam, and AdamW.
Constraint: All generated rules are constrained to a fixed expression tree depth of 5.

Experimental Setup

Benchmarks: The evaluation uses 10 symbolic regression benchmark functions (5 from Hemberg et al. and 5 from Feynman et al.) with varying expression tree depths and complexities.
Architectures: Three distinct fully-connected feed-forward neural network architectures are tested for each benchmark, varying in the number of layers and the use of sigmoid vs. linear activation functions. This results in 30 unique benchmark/network combinations.
Baseline Comparison: Before symbolic regression, a grid search is performed over hyperparameters for established optimizers (Gradient Descent, Heavy-Ball, NAG, AdaGrad, RMSProp, AdaDelta, Adam, AdamW) to establish the best possible baseline performance for each combination.
Evolutionary Process: A Genetic Programming (GP) algorithm evolves populations of candidate update rules.
- Population: Maximum size of 100 candidates per thread.
- Evaluation: Candidates are evaluated by training a randomly initialized neural network for exactly 10 epochs using the candidate rule. The fitness metric is the Mean Squared Error (MSE).
- Seeding: Subsequent experiments attempt to seed populations with rules discovered in previous benchmarks, though they revert to random populations if seeding fails to yield competitive results.
- Termination: Runs stop if improvements become negligible or if 100 expressions outperform the baseline are found.

Key Contributions

Empirical Discovery of Optimizers: The paper demonstrates that symbolic regression can discover explicit weight-update rules that outperform the best hyperparameter-tuned standard optimizers.
Compactness and Interpretability: The discovered rules are structurally compact (fixed depth) and interpretable, often combining recognizable motifs such as adaptive normalization, momentum-like terms, and nonlinear transformations, despite not being explicitly constrained to these forms.
Search Space Exploration: The work explores a vast space of potential equations (estimated at $\approx 1.71 \times 10^{65}$ unique rules for the given depth) to find effective dynamics, revealing that multiple algebraically distinct expressions can achieve comparable performance.

Results

Success Rate: In 25 out of 30 benchmark/neural network combinations, the symbolic regression procedure discovered an update rule that outperformed the best hyperparameter-tuned established optimizer.
Performance Gain: Across the 25 improved cases, the aggregate Mean Squared Error (MSE) reduction was 44.47%.
Failure Cases: The 5 cases where symbolic regression failed to find a superior rule were concentrated in the largest neural network architecture (Neural Network 3), suggesting that the discovered rules may become less competitive as model complexity increases.
Structural Diversity: The best discovered rules did not converge to a single symbolic form. However, four "canonical expression signatures" reappeared across multiple benchmarks, all incorporating accumulated gradient statistics. Many rules utilized nonlinearities (trigonometric, hyperbolic, exponential) and rational expressions involving moving averages.

Significance and Limitations

The authors position this work as a preliminary study rather than a definitive benchmark for deep learning optimizers.

Significance: The results suggest that symbolic regression serves as a lightweight mechanism for discovering compact, explicit, and high-performing optimizer variants. It highlights that effective optimization dynamics may admit low-complexity symbolic representations even when their functional forms vary significantly across tasks.
Limitations: The study is restricted to small neural networks trained for only 10 epochs on symbolic regression tasks, not large-scale machine learning datasets or classification tasks. The search is stochastic, and the discovered rules lack theoretical convergence guarantees.
Future Outlook: The paper concludes that further validation is required on larger architectures, longer training horizons, and standard deep-learning benchmarks. Future work should also incorporate complexity penalties, stability constraints, and post-hoc algebraic simplification to ensure discovered optimizers are both performant and analyzable.