📊 statistics

Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks

This paper characterizes the Reproducing Kernel Hilbert Spaces (RKHS) and sample path smoothness of Neural Tangent Kernels (NTK) and Neural Network Gaussian Processes (NNGP) for a broad class of non-smooth activation functions, revealing how their non-smoothness at zero determines the depth-dependency of the resulting kernels.

Original authors: David Holzmüller, Max Schölpple

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: David Holzmüller, Max Schölpple

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a massive, complex orchestra performs. You can’t listen to every single musician at once, so you look at the "sound" they produce as a whole.

In the world of Artificial Intelligence, a Neural Network is that orchestra. The individual musicians are the "neurons," and the "music" they play is the prediction the AI makes.

For a long time, mathematicians have been trying to study these orchestras by looking at their "Sheet Music"—a mathematical blueprint called a Kernel. This blueprint tells us how the orchestra will behave as it gets bigger and bigger.

The Problem: The "ReLU" Monopoly

Until now, most of our mathematical theories were built around one specific type of musician: the ReLU musician. ReLU is like a musician who only plays if the note is above a certain volume; if it’s too quiet, they stay silent. Because ReLU is so simple (it has a sharp "on/off" switch), the math is easy.

But real-world AI doesn't just use ReLU. It uses more sophisticated "musicians" (activation functions) like SELU, ELU, or GELU. These musicians are smoother, more nuanced, and don't just snap from silence to sound; they fade in and out.

The problem? Because these musicians are more complex, our old "Sheet Music" (the math) didn't work for them. It was like trying to use a blueprint for a drum kit to understand a violin.

The Breakthrough: The "Smoothness" Rule

This paper, written by David Holzmüller and Max David Schölpple, provides the missing blueprints. They discovered that the "sound" of the orchestra depends almost entirely on one thing: How smooth is the musician?

They categorize musicians by their "Smoothness Score":

The Sharp Switch (Low Smoothness): Like ReLU. They have a sudden, jagged jump from zero to one. This creates a specific, predictable type of "sound" (mathematically called a Sobolev Space).
The Smooth Slider (High Smoothness): Like Sigmoid or Tanh. They transition gracefully. This creates a much more refined, "silky" sound.
The Mathematician (Polynomials): These are musicians who only play specific, predictable patterns. Their "sound" is very limited and doesn't get more complex no matter how many musicians you add.

Why does this matter? (The "Depth" Mystery)

The researchers answered a big question: Does adding more layers (making the orchestra deeper) actually change the music?

For most musicians: They found that once you have a few layers, adding more doesn't actually change the type of music being played. It’s like adding more violins to a violin section—the "texture" of the sound stays the same. This explains why some deep AI models behave very similarly to shallower ones.
For the Mathematicians: If you use polynomial musicians, adding layers does change the music. The complexity grows exponentially with every layer.

The "Magic" of Randomness

Finally, they looked at the orchestra at the very moment they start playing (at "initialization"). They found a strange, beautiful phenomenon: even if the musicians are a bit "clunky" or "jagged," when you put thousands of them together, the resulting sound is actually smoother than any individual musician.

It’s like a thousand people whispering at once; the individual whispers might be sharp and breathy, but the collective sound is a smooth, continuous hum.

Summary in a Nutshell

This paper gives scientists the mathematical "tuning forks" they need to study modern, complex AI. Instead of just understanding the "on/off" world of ReLU, we can now mathematically predict how the "smooth and curvy" world of modern AI will learn, generalize, and behave.

Technical Summary: Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks

1. Problem Statement

The theoretical understanding of deep learning often relies on the "kernel regime," where infinitely wide neural networks are modeled using the Neural Tangent Kernel (NTK) (governing training dynamics) and the Neural Network Gaussian Process (NNGP) (governing initialization).

While significant progress has been made in characterizing these kernels for ReLU (and its powers) and infinitely smooth activation functions, a massive gap exists for the most commonly used activations in modern deep learning, such as SELU, ELU, LeakyReLU, GELU, and SiLU. These functions are typically "semi-smooth"—they are infinitely smooth everywhere except at zero. The authors aim to bridge this gap by characterizing the Reproducing Kernel Hilbert Spaces (RKHS) and the smoothness of the resulting functions for this broader class of activations.

2. Methodology

The authors employ a sophisticated analytical framework to study dot-product kernels on the unit sphere $\mathbb{S}^d$ . Their approach follows these technical steps:

Dual Activation Analysis: They leverage the concept of "dual activations" $\hat{\phi}$ , which describe the infinite-width behavior of a single hidden layer. The kernel of a deep network can be expressed recursively through these dual activations.
Boundary Behavior Characterization: The core of the proof relies on analyzing the asymptotic behavior of the dual activation function $\hat{\phi}(t)$ as $t \to 1$ (which corresponds to the behavior of the original activation $\phi$ near its non-smooth point at $0$). They introduce a specialized calculus for function classes ( $Q_{\alpha, \beta}$ ) that handle sums of powers and negligible remainder terms.
Eigenvalue Asymptotics: By studying the decay of the eigenvalues $\mu_l$ of the integral operator associated with the kernel, they connect the kernel's properties to Sobolev spaces $H^s(\mathbb{S}^d)$ .
Recursive Decomposition: They develop a recursive method to propagate the smoothness/non-smoothness through the layers of the network, accounting for different initialization schemes (bias-free, zero-initialized bias, or random bias).

3. Key Contributions

General Characterization of RKHS: The paper provides a definitive characterization (Theorem 9) of the RKHS for both NTK and NNGP for any activation function that is infinitely smooth except at zero.
Unified Theory of Smoothness: They prove that for a wide class of activations, the RKHS is equivalent to a Sobolev space, where the order of the space depends directly on the degree of non-smoothness ( $s$ ) of the activation function.
Parity and Bias Analysis: They identify a "parity phenomenon" in bias-free networks, where the even/odd parts of the activation function dictate whether the kernel contains only even or only odd functions.
Sample Path Smoothness: They derive results (Theorem 14) connecting the RKHS to the actual smoothness of the functions sampled from the NNGP at initialization.

4. Key Results

The results can be summarized by the "smoothness" $s$ of the activation function $\phi$ :

Activation Type	Smoothness ( $s$ )	NNGP RKHS ( $H^s$ )	NTK RKHS ( $H^s$ )
Discontinuous (e.g., Heaviside)	$s=0$	$H^{d/2 + 2^{1-L}}$	Not well-defined
Finite Smoothness (e.g., ReLU, ELU, SELU)	$1 \leq s < \infty$	$H^{d/2 + s + 1/2}$	$H^{d/2 + s - 1/2}$
Infinitely Smooth (e.g., GELU, Tanh)	$s = \infty$	Contained in all $H^t$	Contained in all $H^t$
Polynomial (e.g., $x^m$ )	$s = \infty$	Finite-dimensional (Polynomials)	Finite-dimensional (Polynomials)

Major Findings:

Depth Independence: For most non-polynomial, finitely smooth activations (like ReLU or ELU), the RKHS is equivalent across different depths ( $L \geq 3$ ). This suggests that, in the kernel regime, increasing depth does not change the fundamental function space the network can represent.
Polynomial Exception: Unlike other activations, polynomial activations produce RKHSs that do depend on depth, as the degree of the polynomials grows exponentially with the number of layers.
Smoothness Gain: They show that NNGP sample paths (the functions at initialization) are actually half an order smoother than the RKHS would suggest, a phenomenon analogous to how a random walk converges to Brownian motion.

5. Significance

This paper is a significant theoretical advancement because it moves deep learning theory away from the "ReLU-only" paradigm.

Theoretical Integration: It provides a unified mathematical language to compare different activation functions, allowing researchers to understand how choosing SELU over ReLU affects the complexity of the functions the network can learn.
Generalization and Training: By characterizing the RKHS, the work provides tools to study generalization bounds and spectral bias (the tendency of networks to learn low-frequency functions first) for a much wider array of practical architectures.
Foundation for Future Work: The methodology (handling non-smoothness at zero) provides a roadmap for extending kernel theory to more complex architectures like Residual Networks and Transformers.