🤖 machine learning

TESLA: Taylor Expansion of Sinusoidal Learnable Activations

The paper introduces TESLA, a novel activation function based on learnable sinusoidal combinations that theoretically controls polynomial degree to enhance generalization and robustness, demonstrating superior performance on challenging parity and Forrelation tasks as well as standard vision benchmarks like ImageNet-100.

Original authors: Daehwa Ko, Jaehyeon Kim, Seunghyun Ham, Jay Hoon Jung

Published 2026-08-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daehwa Ko, Jaehyeon Kim, Seunghyun Ham, Jay Hoon Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Puzzle of the Whole Picture

Imagine you are trying to teach a robot to recognize a face. You show it a picture, and it looks at the nose, then the eyes, then the mouth. Standard computer brains, called neural networks, are incredibly good at this kind of "local" detective work. They use simple, step-like switches (like turning a light on or off) to figure out small details. This works wonders for things like spotting a cat in a photo or translating a sentence, where the answer depends on nearby clues.

However, some puzzles are trickier. Imagine a game where you have a long row of light switches, and the answer depends on the entire row at once: "Is the total number of switches turned on odd or even?" To solve this, the robot can't just look at one switch or even a small group; it has to count them all together. This is a "global" problem. Traditional robot brains struggle here because they are wired to look at small, local pieces first. They often get stuck, missing the big picture because they are too focused on the tiny details. Scientists have been trying to build a robot brain that can naturally understand these big, whole-picture patterns without needing to be incredibly huge or deep. This is where a new idea called TESLA comes in.

The Magic of the Musical Activation

The researchers behind this study, working at Korea Aerospace University, propose a new way for these robot brains to "think." They call their invention TESLA (Taylor Expansion of Sinusoidal Learnable Activations). To understand what they did, let's look at how a standard robot neuron works. Usually, it takes an input, runs it through a simple, rigid switch, and passes the result along. It's like a door that only opens if you push hard enough, but it doesn't really "sing" or change its tune.

TESLA replaces this rigid switch with a musical instrument. Instead of a simple on/off button, the neuron uses a mix of sine and cosine waves—think of these as smooth, rolling waves like sound or ocean tides. But here is the magic: TESLA doesn't just use one wave. It builds a custom song by mixing together many different waves, each with its own volume knob. The robot learns to turn these knobs up or down during training.

Why does this matter? In the world of math, these waves can be combined to create shapes that look like complex polynomials (curves with many bumps). By adjusting the knobs, the robot can decide exactly how "wiggly" or complex its thinking should be. If the problem is simple, it uses smooth, low-frequency waves. If the problem is a tricky global puzzle (like the odd-or-even switch game), it can crank up the high-frequency waves to capture those complex, whole-picture patterns that standard robots miss.

Solving the "Odd or Even" Mystery

The team tested this idea on a famous difficult puzzle called the parity problem. Imagine a string of 32 switches. The robot must guess if the total number of "on" switches is odd or even. This is a nightmare for standard robots because the answer depends on every single switch interacting with every other one.

In their experiments, the researchers trained their TESLA-powered robots with 100,000 examples. Even though there are over 4 billion possible combinations of 32 switches (meaning the robot only saw a tiny, tiny fraction of the possibilities), the TESLA robot learned the rule incredibly well. It achieved near-perfect accuracy on clean data. Even more impressive, when the researchers started messing with the data—intentionally lying to the robot by flipping 30% of the answers—the TESLA robot stayed strong and kept guessing correctly.

In contrast, the "standard" robots using old-fashioned switches (like ReLU) or even other wave-based robots (like SIREN) completely gave up. As the number of switches increased, their accuracy dropped to about 50%, which is no better than just guessing "heads or tails" randomly. The TESLA robot, however, could see the global pattern even when the data was noisy and messy.

From Math Puzzles to Real-World Photos

The researchers didn't stop at math puzzles. They wanted to see if this "musical" thinking could help with real-world tasks, like recognizing images. They tested TESLA on a dataset of 100 different types of images (ImageNet-100), using popular robot brain architectures like ResNet and Vision Transformers.

The results were promising. The TESLA robots performed just as well as the standard robots at identifying images, but with a crucial difference: stability. When they tried to use other wave-based methods (like SIREN) on these large image tasks, the robots became unstable and failed to learn. TESLA, thanks to its careful design of the "volume knobs" (which the authors call a coefficient budget), stayed calm and learned effectively. It managed to do this without slowing down the robot or making it much bigger; the extra work required was negligible, about 1% more than usual.

What This Means for the Future

The paper suggests that by giving robot neurons the ability to tune their own "frequency" and complexity, we can help them solve problems that require looking at the whole picture rather than just the parts. The authors show that this approach is not just a theoretical trick but a practical tool that works on everything from solving physics equations to recognizing faces.

While the paper proves that TESLA works exceptionally well on these specific tests and suggests it could be a powerful replacement for standard switches in many areas, the researchers are careful to note that there is still more to explore. They plan to investigate how to make these musical neurons even faster on computer chips and how they might help large language models understand long, complex stories. For now, though, TESLA stands as a clever new way to teach machines to hear the music in the data, rather than just counting the notes.

Technical Summary: TESLA (Taylor Expansion of Sinusoidal Learnable Activations)

1. Problem Statement

Standard neural networks, which typically rely on stacking local nonlinearities (e.g., ReLU, GeLU) with linear maps, exhibit a "spectral bias" toward learning low-frequency (low-degree) components. This limitation makes them inefficient for tasks requiring global interactions, high-order combinations of input coordinates, or specific global symmetries.

A canonical example of this challenge is the parity problem (determining if the number of ones in a binary vector is odd or even). This task is linearly inseparable and requires global interaction across all input bits. Traditional architectures often fail to generalize to parity tasks, especially as input dimensionality ( $n$ ) increases or when data is scarce, because representing such high-order structures typically demands excessive depth, width, or weight norms. Furthermore, existing attempts to address spectral bias—such as periodic activations (SIREN) or Fourier feature embeddings—often suffer from optimization instability, overfitting in low-data regimes, or a lack of explicit control over the polynomial degree of the activation.

2. Methodology: TESLA

The authors propose TESLA (Taylor Expansion of Sinusoidal Learnable Activations), a parametric activation function designed to provide explicit, learnable control over the polynomial degree and frequency components of neural network activations.

2.1 Definition

TESLA is defined as a finite linear combination of sine and cosine terms shared across all neurons in a layer:
$\phi_K(z) = \sum_{k=1}^{K} \left( \frac{a_k}{k} \sin(kz) + \frac{b_k}{k} \cos(kz) \right)$
where:

$K \in \mathbb{N}$ controls the maximum frequency (harmonic count).
$\{a_k, b_k\}_{k=1}^K$ are learnable scalar coefficients.
The $1/k$ scaling is a design choice to manage gradient magnitudes.

2.2 Theoretical Interpretation

While implemented as a trigonometric expansion, the authors analyze TESLA via its Taylor (Maclaurin) expansion. This reveals that the effective polynomial coefficients of the activation are explicit linear functionals of the learned trigonometric coefficients $\{a_k, b_k\}$ . This formulation allows the network to selectively amplify or attenuate specific polynomial orders (degrees) at the activation level, rather than relying on input-side spectralization (like Fourier features) which leaves mode selection to later layers.

2.3 Stability and Complexity Control

To ensure stability and prevent overfitting, TESLA introduces a coefficient budget:
$A_K = \sum_{k=1}^{K} (|a_k| + |b_k|)$
Theoretical analysis shows that $A_K$ bounds the Lipschitz constant of the activation and the network. Specifically, the Lipschitz constant of an $L$ -layer network is bounded by the product of layer operator norms and $A_K^{L-1}$ . This budget allows for the control of gradient magnitudes and Rademacher complexity, ensuring that increasing the frequency range ( $K$ ) does not inherently destabilize optimization.

3. Key Contributions

The paper presents three primary contributions:

Theoretical Guarantees: The authors derive tight Lipschitz bounds and Rademacher complexity-based generalization guarantees. They characterize the learning dynamics using Neural Tangent Kernel (NTK) analysis, showing that TESLA's inductive bias naturally favors the recovery of low- and mid-frequency structures while allowing for the selective learning of higher-order components.
Constructive Approximation Results: The paper provides upper bounds for approximating ridge- and interaction-type functions using TESLA. Conversely, it sketches lower bounds demonstrating that standard piecewise-linear activations require substantially more resources (depth/width) to represent oscillatory or global structures compared to TESLA.
Empirical Validation: The authors demonstrate that TESLA generalizes effectively on synthetic benchmarks (Parity, Forrelation) and continuous-domain tasks (PINNs, INRs), while remaining competitive on large-scale vision tasks (ImageNet-100) with negligible computational overhead.

4. Experimental Results

4.1 Synthetic Benchmarks

Parity Task: On parity tasks with input length $n=32$ , TESLA achieves strong generalization using only 100,000 training samples (approx. 0.002% of the input space). Crucially, it remains robust under heavy label noise (up to 30%), retaining high accuracy where baseline activations (ReLU, GeLU) collapse to random guessing (50%).
Comparison with Periodic Baselines: TESLA significantly outperforms SIREN, SNAKE, and Fourier feature embeddings on parity and Forrelation tasks. While SIREN and Fourier embeddings often fail to capture global correlations in high-noise or high-dimension regimes, TESLA maintains stable performance. For example, on 32-bit parity with no noise, TESLA achieves ~96% accuracy, whereas SIREN drops to ~50%.
Forrelation: On the Forrelation task (a global correlation decision problem), TESLA converges faster and achieves higher accuracy across various hidden layer sizes compared to standard activations.

4.2 Continuous-Domain Tasks

PINNs: For solving the 1D viscous Burgers' equation, TESLA achieves the lowest final loss among compared activations (including ReLU, Tanh, and SIREN).
Implicit Neural Representations (INRs): On image reconstruction tasks (Kodak24, DIV2K), TESLA reaches target PSNR thresholds faster and achieves higher final quality (PSNR/SSIM) than SIREN and ReLU.
Mixed-Frequency Regression: TESLA is the only method tested that achieves a test MSE below 0.011 for signals composed of mixed high-frequency components.

4.3 Large-Scale Vision (ImageNet-100)

TESLA was evaluated on ImageNet-100 using ViT, MLP-Mixer, ResNet-18, and MobileNetV3 architectures.

Performance: TESLA achieves Top-1 accuracy comparable to or slightly better than ReLU, GeLU, and SiLU across all architectures.
Efficiency: The method incurs negligible throughput overhead (approx. 1% increase in FLOPs).
Stability: Unlike SIREN, which exhibited severe optimization instability and poor accuracy (e.g., ~4% Top-1 on ViT) under the same training settings, TESLA trained stably and reached standard accuracy levels.

5. Significance and Claims

The paper claims that TESLA offers a robust, practical alternative to input-only spectralization methods. By moving spectral control to the activation level via learnable coefficients, TESLA allows practitioners to tune the inductive bias of a network with minimal architectural changes while preserving compatibility with standard training pipelines.

The authors emphasize that TESLA bridges the gap between theoretical spectral bias analysis and practical application. It enables the explicit control of polynomial degrees, facilitating the learning of global, high-order interactions that standard activations struggle with, without sacrificing stability or scalability on real-world vision workloads. The work suggests that activation-level degree control is a viable strategy for improving sample efficiency and generalization in tasks dominated by global symmetries.