📊 statistics

Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information

This paper identifies that the major eigenspaces of the Fisher information matrix for 2-layer ReLU networks with random hidden weights correspond to function spaces spanned by spherical harmonic functions of order at most 2, which collectively account for over 97% of the matrix's trace.

Original authors: Ka Long Keith Ho, Yoshinari Takeishi, Junichi Takeuchi

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ka Long Keith Ho, Yoshinari Takeishi, Junichi Takeuchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, complex machine—a two-layer neural network with a "ReLU" activation function (think of it as a machine that only turns on when a signal is strong enough). This machine has a hidden layer with thousands of tiny gears (neurons) that are set up randomly and never moved. Only the final layer of gears is adjustable.

The authors of this paper wanted to understand: When we train this machine, what specific patterns or shapes does it learn first?

To answer this, they didn't just watch the machine learn; they looked at its "blueprint" using a mathematical tool called the Fisher Information Matrix. Think of this matrix as a map that shows which directions in the machine's "learning space" are the easiest to move in. Just like a hilly landscape has steep slopes and flat valleys, this map has "easy paths" (large eigenvalues) and "hard paths" (small eigenvalues).

The paper makes a fascinating discovery: 97.7% of the machine's learning power is concentrated in just three specific "directions" or "modes." No matter how huge the machine gets, it almost entirely ignores the other 2.3% of possibilities.

Here is what those three main "modes" actually look like, explained simply:

1. The "Distance" Mode

The very first and strongest pattern the machine learns is simply how far a point is from the center.

The Math: It learns a function proportional to $\|x\|$ (the length of the input vector).
The Analogy: Imagine you are standing in a dark room with a flashlight. The first thing the machine learns is how bright the light is based on how far you are from the center of the room. It doesn't care about where you are (left or right), only how far you are.

2. The "Coordinate" Mode

The second group of patterns (there are $d$ of them, where $d$ is the number of input dimensions) learns about individual directions.

The Math: It learns functions proportional to $x_l$ (the value of a specific coordinate).
The Analogy: Now the machine learns to tell you if you are moving North, South, East, or West. It breaks the world down into simple, straight lines. If you move along the "x-axis," it notices that specifically.

3. The "Interaction" Mode

The third and largest group of patterns learns about how different directions interact with each other, but in a very specific way.

The Math: It learns functions proportional to $x_\alpha x_\beta / \|x\|$ .
The Analogy: This is like noticing that moving North and East at the same time creates a specific diagonal effect, but the machine "normalizes" it by the total distance. It's learning about the shape of the interaction between two directions, rather than just the directions themselves.

Why Does This Matter?

The paper claims that because the "easy paths" (the three modes above) are so dominant, gradient descent (the algorithm that trains the machine) will naturally rush to learn these three things first. It's like a ball rolling down a hill; it will naturally roll down the steepest, widest valleys first (the Distance and Coordinate modes) before it even thinks about the tiny, narrow cracks in the rock (the other 2.3% of patterns).

The "Spherical" Connection

The authors point out that these three patterns are actually related to Spherical Harmonics.

The Analogy: Imagine the surface of a basketball. Mathematicians have a special set of "musical notes" (spherical harmonics) that can describe any vibration on that ball.
- The "Distance" mode is like the ball vibrating as a whole (the lowest note).
- The "Coordinate" modes are like the ball vibrating in simple up-down or left-right waves.
- The "Interaction" modes are like more complex, twisting waves on the surface.
  The paper shows that this random neural network is essentially "playing" the first few notes of this spherical song.

What They Tested

The authors ran computer simulations to prove this. They built these random machines with different sizes and checked if the output actually matched the simple formulas they derived (like "Distance" or "Coordinate").

The Result: The math held up perfectly. As they made the hidden layer bigger (more gears), the machine's behavior got closer and closer to the simple formulas. The error (difference between the machine and the formula) got smaller and smaller.

The Catch (Limitations)

The paper is very careful to note that this only works under specific conditions:

Random Inputs: The data the machine is trained on must look like a standard "bell curve" (Gaussian distribution), like throwing darts at a board where most land in the center. If the data is weird or clustered in a specific shape, these simple rules might break.
Infinite Size: The theory assumes the hidden layer is infinitely large. In real life, with smaller machines, the results are an approximation, though the simulations show it works well even with reasonably large sizes.

In summary: This paper reveals that a random, wide neural network isn't a chaotic mess. It has a very clear, simple "voice." When it starts learning, it almost exclusively sings three types of songs: "How far am I?", "Which way am I going?", and "How do these two directions mix?" Everything else is just background noise.

Technical Summary: Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information

Problem Statement
This paper investigates the function space represented by a two-layer, bias-free ReLU neural network with randomly fixed hidden weights and trainable output weights. Specifically, the authors aim to identify the specific functional forms corresponding to the dominant eigenspaces of the Fisher Information Matrix (FIM). Previous work by Takeishi et al. [7, 8, 9] established that for such networks, the eigenvalue distribution of the FIM concentrates heavily on the first three eigenspaces, which collectively account for approximately 97.7% of the matrix's trace, regardless of the number of hidden units ( $m$ ). While the spectral properties were known, the specific function spaces induced by these major eigenspaces remained unidentified.

Methodology
The study analyzes a model where the input $x \in \mathbb{R}^d$ follows a standard Gaussian distribution $N(0, I_d)$ , and the output is $y = f_v(x) + \epsilon$ , with $f_v(x) = X(x)^\top v$ . Here, $X(x) = \sigma(x^\top W)$ represents the hidden layer output using the ReLU activation $\sigma(\cdot)$ , where $W \in \mathbb{R}^{d \times m}$ contains i.i.d. Gaussian weights $N(0, 1/m)$ , and $v \in \mathbb{R}^m$ is the trainable output weight vector.

The methodology proceeds as follows:

Spectral Analysis: The authors utilize the approximate spectral decomposition of the FIM, $J = \mathbb{E}_{x}[X(x)X(x)^\top]$ $J = E_{x} [X (x) X (x)^{⊤}]$ , derived in prior literature. They focus on the three dominant eigenspaces corresponding to the largest eigenvalues:
- The first eigenspace (multiplicity 1) with eigenvalue $\approx (2d+1)/4\pi$ .
- The second eigenspace (multiplicity $d$ ) with eigenvalue $1/4$ .
- The third eigenspace (multiplicity $d(d+1)/2 - 1$ ) with eigenvalue $\approx 1/(2\pi(d+2))$ .
Function Space Mapping: By defining the inner product in the function space as $\langle f, g \rangle = \mathbb{E}_{x \sim N(0, I_d)}[f(x)g(x)]$ , the authors map the approximate eigenvectors of the parameter space ( $v$ ) to their corresponding asymptotic functional forms in the limit as $m \to \infty$ .
Asymptotic Derivation: The paper derives the limit of $X(x)^\top v$ for the specific eigenvectors associated with the three dominant clusters, proving convergence in probability to specific functions involving spherical harmonics.
Validation: The theoretical results are validated through numerical simulations across varying input dimensions ( $d$ ) and hidden layer sizes ( $m$ ), measuring the Mean Absolute Error (MAE) between the realized network outputs and the derived asymptotic limits.

Key Contributions and Results
The primary contribution is the identification of the specific basis functions that span the function space corresponding to the 97.7% of the FIM's trace. As $m \to \infty$ , the network outputs corresponding to the three dominant eigenspaces converge to the following forms:

First Eigenspace: The function converges to a radial component proportional to the Euclidean norm:
$F_0(x) \propto \|x\|$
Second Eigenspace: The functions converge to coordinate-specific linear terms:
$F_l(x) \propto x_l \quad \text{for } l \in \{1, \dots, d\}$
Third Eigenspace: The functions converge to terms involving products of coordinates normalized by the norm, specifically related to spherical harmonics of degree 2:
$F_{\alpha\beta}(x) \propto \frac{x_\alpha x_\beta}{\|x\|} \quad (\alpha < \beta)$
$F_\gamma(x) \propto \frac{x_\gamma^2}{\|x\|} - \text{correction terms} \quad (\gamma \in \{1, \dots, d-1\})$

The authors demonstrate that these functions form an approximately orthogonal basis in the function space under the Fisher metric. The simulation results confirm that as $m$ increases, the Mean Absolute Error between the network's realization and these theoretical limits decreases, validating the asymptotic approximations.

Significance and Claims
The paper claims that these results provide a rigorous characterization of the "learning dynamics" of simple ReLU networks in the infinite-width limit. By linking the FIM's spectral decomposition to specific functional forms, the authors argue that:

Gradient Descent Prioritization: Since gradient descent converges faster along directions corresponding to larger eigenvalues of the Hessian (which approximates the FIM near the optimum), the network prioritizes learning features proportional to $\|x\|$ , followed by $x_l$ , and then the quadratic terms $x_\alpha x_\beta / \|x\|$ .
Connection to Neural Tangent Kernels (NTK): The results are interpreted as identifying the eigenvalues and eigenfunctions of the first three degrees of the Mercer decomposition of the Neural Tangent Kernel for two-layer networks. This connects the finite-width FIM analysis to the infinite-width NTK literature, specifically relating to the work of Bietti and Ma [3].
Function Space Dynamics: The work offers a framework for understanding how overparameterized networks express functions, suggesting that the initial training phase is dominated by low-degree spherical harmonic functions (orders 0, 1, and 2).

Limitations
The authors explicitly note several limitations:

Input Distribution: The orthogonality and specific functional forms rely heavily on the input $x$ being rotationally invariant (specifically standard Gaussian). Non-Gaussian or non-rotationally invariant distributions may alter the hierarchy of the basis functions.
Finite Width: The analysis assumes the infinite-width limit ( $m \to \infty$ ). Finite-width networks may exhibit deviations due to stochastic effects.
Data Manifolds: The current framework does not address data distributed on lower-dimensional manifolds, where the FIM might become degenerate.
Architecture: The study is restricted to two-layer, bias-free ReLU networks; extensions to deeper architectures or different activation functions are not covered.