🤖 machine learning

Fractional Optimizers Meet Fractal Activation Functions: An Empirical Study of Multi-Scale Optimization in Neural Network

This empirical study investigates the interaction between fractional optimizers and fractal activation functions, revealing that while neither serves as a universal replacement, specific pairings—such as regularization-style fractional scaling with selected fractal activations—offer promising improvements for neural network training.

Original authors: Sebastian Raubitzek, Georg Goldenits, Sebastian Schrittwieser, Philip König, Kevin Mallinger

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sebastian Raubitzek, Georg Goldenits, Sebastian Schrittwieser, Philip König, Kevin Mallinger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, computers learn by adjusting their internal settings to minimize errors, a process driven by mathematical tools called optimizers. These tools act like a hiker trying to find the lowest point in a vast, foggy valley, taking steps based on the slope of the ground beneath their feet. For decades, the standard approach has been to look only at the immediate slope, ignoring what happened a moment ago. However, nature is rarely so simple. Many natural patterns, from the jagged edge of a coastline to the fluctuating rhythm of a heartbeat, possess a rough, self-similar complexity that repeats at every scale. Mathematicians call this fractal geometry. Recently, researchers have begun to wonder if these two ideas—rough, multi-scale patterns and the mathematical tools used to measure them—could be combined to help computers learn better. Specifically, they asked whether using "fractal" activation functions, which inject this complex, rough structure into the neural network, would work better when paired with "fractional" optimizers, which are designed to remember past steps over a longer history rather than just reacting to the present.

A team of researchers set out to test this idea by building a unified experimental framework to see how these two concepts interact. They did not simply guess; they constructed a rigorous test involving two distinct stages. First, they created controlled, two-dimensional landscapes that mimicked the roughness of fractal geometry. Some of these landscapes were smooth and predictable, while others were deliberately scrambled with layers of tiny, repeating oscillations to simulate the complexity of real-world data. They sent twenty-one different optimization methods across these surfaces to see which could find the lowest points most reliably. In the second stage, they moved to a more realistic setting, training neural networks on ten different public datasets used for classification tasks. They paired these networks with four specific types of fractal activation functions and tested them against the same twenty-one optimizers, running each experiment forty times to ensure the results were not just lucky accidents.

The results revealed a surprising truth about how these tools work together. On the controlled, rough landscapes, the methods that remembered their past steps performed exceptionally well. When the terrain was filled with the kind of persistent, repeating patterns that fractals create, optimizers that looked back at their history could navigate the noise and find the target more effectively than those that only looked at the immediate slope. However, the story changed dramatically when the researchers moved to the neural network experiments. In the chaotic environment of training a real computer model, where data is processed in small, noisy batches, the same memory-based methods often struggled. The very mechanism that helped on the smooth, deterministic surfaces—looking back at previous steps—tended to amplify the random noise found in real-world data, leading to instability or slower progress.

The study found that the most successful approach was not to blindly apply memory or fractal structures everywhere, but to find specific, careful pairings. The researchers discovered that a particular type of optimizer, which adjusts the size of the learning step based on a mathematical rule without storing a long history of past gradients, proved to be the most robust and effective across almost all datasets. This method, when combined with a specific fractal activation function that adds a moderate amount of roughness to the network, consistently produced the best results. In contrast, the methods that relied heavily on storing and averaging past gradients often performed poorly in the noisy environment of neural network training, despite their success on the controlled surfaces.

Ultimately, the paper demonstrates that while fractal activation functions and fractional optimizers are mathematically compatible and can be powerful when matched correctly, they are not a universal solution. The benefit comes from the specific combination of the activation function, the optimizer's design, and the nature of the data. The researchers showed that simply adding "fractal" or "fractional" labels to a method does not guarantee better performance; in fact, indiscriminate use of memory-based techniques can be detrimental in noisy settings. Instead, the most practical path forward is to treat the choice of activation function and optimizer as a joint decision, selecting the specific pairing that suits the problem at hand. The study concludes that for most practical applications, a method that adjusts step sizes locally without heavy reliance on past history offers the best balance of speed, stability, and accuracy, proving that sometimes, looking back is less useful than looking forward with the right tools.

Technical Summary: Fractional Optimizers Meet Fractal Activation Functions

Problem Statement

Neural network training often encounters loss landscapes with irregular, multi-scale structures. While fractal geometry provides a mathematical framework for describing such roughness, and fractional calculus offers tools for analyzing non-local behavior and memory effects, these two fields have historically developed independently within neural network research.

Previous work established that fractal activation functions (derived from Weierstrass- and Blancmange-type series) can inject multi-scale nonlinear representations into networks, potentially increasing expressivity. Separately, fractional optimizers have been proposed to extend first-order gradient descent by incorporating non-integer order derivatives, which introduce memory effects and non-local behavior. However, the interaction between these two components—specifically, whether optimizers designed with fractional memory principles are well-suited to train networks utilizing fractal activations—remains unexplored. Furthermore, the efficacy of fractional optimizers on objective landscapes explicitly perturbed with fractal structures has not been systematically evaluated.

Methodology

The study employs a two-stage experimental framework to investigate the interaction between fractal activations and fractional optimizers.

1. Mathematical Foundation

The authors establish a theoretical link between Weierstrass-type functions and fractional derivatives. They demonstrate that for a Weierstrass-type function with a roughness exponent $\gamma_H$ , fractional derivatives of order $\nu < \gamma_H$ exist and map the function to another Weierstrass-type function with a reduced exponent. The discrete Grünwald–Letnikov (G-L) formulation is identified as the computational mechanism that discretizes these derivatives, providing a weighted history sum with algebraically decaying coefficients. This kernel serves as the basis for the memory-based optimizers, while the truncated Weierstrass series serves as the basis for the fractal activations.

2. Experimental Design

The study evaluates 21 optimizers across five groups and four fractal activation functions (plus standard baselines) on two distinct tasks:

Task A: Controlled Surface Optimization
- Surfaces: Ackley and Himmelblau benchmark functions.
- Variants: Standard forms and additive variants perturbed by Weierstrass-type series ( $P_J$ ) to introduce controlled multi-scale roughness.
- Metrics: Final loss, best loss, success rate (reaching known minima), distance to minima, and runtime.
- Setup: 40 repeated runs per optimizer per surface variant.
Task B: Neural Network Classification
- Datasets: Ten public classification datasets from OpenML (e.g., Iris, Wine, Ionosphere, Vehicle Silhouettes).
- Architecture: Feed-forward Multilayer Perceptrons (MLPs).
- Activations: Four selected fractal activations (Modulated Blancmange, Decaying Cosine, Modified Weierstrass–Tanh, Weierstrass–Mandelbrot $x+\sin$ ) alongside ReLU and Tanh.
- Setup: 40 repeated runs per configuration (optimizer $\times$ activation $\times$ fractional order).

3. Optimizer Families

The 21 optimizers are categorized as follows:

Classical Baselines: SGD, RMSprop, Adam, Adadelta.
Herrera-Type (Local Scaling): Rescale the current gradient using a Caputo-inspired power law factor $f_\nu(g_t)$ without storing history (e.g., FSGD, FAdam).
Memory-Based (Explicit History): Replace the current gradient with a finite-history G-L gradient $g^{(\nu)}_t$ , stabilized by descent safeguards and norm matching (e.g., MemoryFSGD, MemoryFAdam).
Adaptive Memory-Based: Combine the ordinary gradient and the norm-matched fractional gradient via a bounded, stability- and loss-controlled mixing coefficient $\lambda_t$ (e.g., AdaptiveMemoryFSGD).
Related Work: Includes methods like AdaGL, FCSGD_GL, and AOFGD variants.

Key Contributions

Unified Theoretical Framework: The paper explicitly connects the analysis of fractional derivatives of Weierstrass-type functions with the discrete G-L constructions used in optimizers, demonstrating that fractal activations and fractional optimizers are two realizations of the same multi-scale mathematical mechanism.
New Adaptive Framework: Introduction of the AdaptiveMemory family of optimizers. These methods keep the fractional order $\nu$ fixed but adapt a trust coefficient $\lambda_t$ to control the contribution of the memory term, allowing for an exact reduction to the classical optimizer when memory is deemed unreliable.
Systematic Empirical Comparison: A comprehensive evaluation of 21 optimizers on both fractally perturbed surfaces and neural network classification tasks, providing a granular view of how different fractional mechanisms (local scaling vs. explicit memory) perform under varying conditions.

Results

Surface Optimization Experiments

Optimizer Family Dominance: The base optimizer family (SGD vs. RMSprop/Adadelta) was the dominant factor. SGD-type methods outperformed others on both Ackley and Himmelblau, regardless of fractional modifications.
Effect of Fractal Perturbation:
- On Himmelblau, the additive fractal perturbation reduced the reliability of all methods but favored adaptive-order methods (AOFGD_Adam) and Adam-based variants.
- On Ackley, the perturbation created local low-value regions that memory-based and stochastically perturbed SGD variants (e.g., FCSGD_GL, MemoryFSGD) could exploit, though overall success rates remained low due to the flat outer plateau.
Memory vs. Scaling: On the controlled surfaces, explicit gradient memory (Memory-based group) proved reliable, particularly on perturbed landscapes where persistent oscillatory structures existed.

Neural Network Classification Experiments

Fractal Activations: Fractal activations dominated the top-15 configurations on 8 of the 10 datasets and provided the single best configuration on 6 datasets. However, when averaged across all optimizer configurations, standard activations (ReLU, Tanh) often outperformed fractal ones. The Modified Weierstrass–Tanh (subcritical) was the most consistent fractal activation, while the supercritical Weierstrass–Mandelbrot variant generally performed poorly.
Optimizer Performance:
- Herrera-Type (Local Scaling): These methods were the strongest overall, achieving the sole best configuration on 6 datasets and sharing first place on a 7th. They offered high performance with negligible computational overhead.
- Explicit Memory (GL-Memory): These methods performed poorly in classification, often falling outside the top 15 and exhibiting high variance. The authors attribute this to the amplification of sampling noise in minibatch training, contrasting with their success on deterministic surfaces.
- Adaptive Memory: This group showed improved robustness over plain memory methods, ranking higher in group averages and winning on two datasets (Climate-Model Simulation Crashes, Glass Identification).
Interaction: The best results were achieved through specific pairings (e.g., Modified Weierstrass–Tanh with Herrera-type optimizers) rather than the universal application of fractal components. No single fractional order ( $\nu$ ) was universally optimal; winning configurations used $\nu \in \{0.75, 1.25, 1.50\}$ .

Significance and Claims

The paper claims that fractal activations and fractional optimizers are mathematically compatible modifications of different parts of the training problem (representation vs. optimization dynamics), but their combination yields benefits only through specific, identifiable pairings rather than as universal replacements.

Mechanism vs. Ranking: The study highlights a critical divergence: explicit gradient memory is beneficial on deterministic, multi-scale landscapes (surface experiments) but becomes a liability in stochastic minibatch training (classification), where it amplifies noise.
Practical Guidance: The authors recommend:
1. Jointly selecting the activation function, optimizer formulation, and fractional order.
2. Using Herrera-type local scaling as the default fractional extension due to its negligible cost and exact classical limit.
3. Treating the fractional order as a discrete hyperparameter to be tuned.
4. Reserving explicit gradient memory for low-noise settings with persistent multi-scale structures, or using it only through bounded adaptive mixing (the proposed $\lambda_t$ framework).

The study concludes that while the unified framework provides a reproducible basis for further research, the indiscriminate use of explicit gradient memory is not supported by the data, and the "best" method is highly dependent on the specific dataset and activation-optimizer pairing.