📊 statistics

How does feature learning reshape the function space?

This paper demonstrates that early-stage feature learning in two-layer neural networks fundamentally reshapes the induced function space by transforming the feature distribution into a target-dependent spiked Gaussian, which creates a data-adaptive kernel that selectively amplifies signal-aligned directions and mixes eigenfunctions rather than merely rescaling a fixed kernel.

Original authors: João Lobo, Bruno Loureiro, Long Tran-Than, Fanghui Liu

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: João Lobo, Bruno Loureiro, Long Tran-Than, Fanghui Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize a specific pattern in a massive, chaotic room full of furniture.

The Old Way (Fixed Kernel):
Traditionally, you might give the robot a rigid, pre-made map of the room. This map has fixed rules: "If you see a chair, look here; if you see a table, look there." The robot can only learn by adjusting how much it trusts each part of this fixed map. It's like trying to find a specific book in a library where the shelves never move; you can only change which books you pull off the shelf, not the shelf's position itself. This is what "lazy training" looks like in neural networks—the features (the map) stay frozen.

The New Way (Feature Learning):
Modern neural networks are different. They don't just adjust the weights on a fixed map; they actually move the shelves. They learn to rearrange the room itself to make the target pattern easier to find. This paper asks: How exactly does the robot move the shelves when it takes its first big step to learn?

Here is the breakdown of what the paper discovered, using simple metaphors:

1. The "Spiked" Map

The researchers looked at what happens after the robot takes one giant step of learning (a "gradient step"). They found that the robot's internal map of the room changes in a very specific way.

Imagine the room is a giant, flat field of grass (representing random data). Before learning, the robot sees the grass as perfectly uniform. After one big learning step, the robot suddenly sees a giant, glowing spike sticking out of the ground in the exact direction of the answer it's looking for.

The paper proves that this new map isn't just a slightly tweaked version of the old one. It's as if the robot's perception has been warped by a "target-dependent" force. It's like putting on special glasses that make the direction of the answer look huge and bright, while everything else stays relatively normal.

2. Reshaping the "Function Space"

In math terms, the paper talks about "function space." Let's call this the Playground of Possibilities.

Before learning: The playground is a flat, featureless field. Any path you take is just as likely as any other.
After learning: The playground gets reshaped. The ground tilts. The paths that lead toward the answer become smooth, wide highways. The paths that lead away from the answer become steep, rocky cliffs.

The paper shows that this reshaping isn't random. It specifically amplifies the "directions" in the data that match the answer (the "signal") and mixes different types of patterns together to make them easier to learn.

3. The "Mixing" Effect

One of the most interesting findings is how the robot combines different types of patterns.
Imagine the robot is trying to learn a complex shape (like a curve).

Old view: The robot might try to learn the "straight line" part and the "curve" part separately.
New view (Feature Learning): The paper shows that the robot's learning process mixes the straight line with the curve. It creates a new, hybrid pattern that is perfectly tuned to the answer.

Specifically, for the common "ReLU" activation (a standard tool in AI), the robot takes the simplest pattern (a straight line pointing at the answer) and mixes it with a slightly more complex pattern (a curve). This creates a super-pattern that is much better at solving the problem than either part alone.

4. The Size of the Step Matters

The paper also found that how big the robot's learning step is changes the result.

Small steps: The robot makes tiny adjustments. The map changes a little, but it's mostly the same old map.
Big steps: The robot makes a giant leap. This causes a massive reshaping of the playground. The "spike" in the direction of the answer becomes very prominent, and the mixing of patterns happens much more aggressively.

The Bottom Line

The paper concludes that feature learning isn't just about turning up the volume on a fixed radio station (rescaling a fixed kernel). Instead, it's like rewiring the radio itself to tune into a new frequency that matches the music you want to hear.

By taking a single, large step, the neural network fundamentally alters its internal landscape. It warps its perception of the data to highlight the specific features that matter, effectively "skipping" the early, slow phases of learning by immediately creating a custom-made map for the problem at hand.

Technical Summary: How Feature Learning Reshapes the Function Space

1. Problem Statement

Modern neural networks (NNs) are distinguished from fixed-kernel methods (such as Random Feature Models or lazy training regimes) by their ability to perform feature learning—adapting internal representations to the data structure during training. While the success of NNs is often attributed to this mechanism, the precise impact of feature learning on the induced function space remains poorly understood. Existing characterizations of kernels and neural networks are largely static; they describe the function space at initialization or in the infinite-width limit but fail to explain how training algorithms, specifically gradient descent, dynamically reshape this space.

This work addresses the question: How does the kernel (and the associated function space) evolve under gradient updates, and what information is progressively learned? The authors focus on the early stages of training a two-layer neural network on a Gaussian single-index model, a setting known to capture the most general class of functions learnable in the high-dimensional proportional regime.

2. Methodology

The authors analyze the evolution of the function space following a single large gradient descent step on the first-layer weights of a two-layer network. The setup involves:

Data Model: A Gaussian single-index model $y_i = g(\langle w^*, x_i \rangle) + \epsilon_i$ , where $x_i \sim \mathcal{N}(0, I_d)$ .
Network Architecture: A two-layer network $f(x) = \frac{1}{\sqrt{m}} \sum_{j=1}^m a_j \sigma(w_j^\top x)$ with ReLU or general Lipschitz activations.
Training Regime: The "proportional regime" where the number of samples $n$ , hidden units $m$ , and dimension $d$ grow to infinity at fixed ratios ( $\alpha = n/d, \beta = m/d$ ). The step size $\eta$ scales as $\Theta(d^\zeta)$ with $\zeta \in [1/2, 1)$ , allowing for significant weight updates.
Two-Stage Training:
1. Feature Learning: A single gradient step updates the first-layer weights $W$ while keeping the second layer $a$ fixed.
2. Linear Readout: The second-layer weights are optimized via ridge regression on a fresh set of samples.

The core analytical approach involves characterizing the post-update feature distribution and deriving the resulting data-adaptive kernel. The authors utilize random matrix theory, spectral analysis of integral operators, and Taylor-like expansions to relate the updated kernel to the initial isotropic kernel.

3. Key Contributions and Results

A. Approximation by a Target-Dependent Spiked Gaussian

The paper proves that after a large gradient step, the distribution of the learned features is well-approximated by a target-dependent spiked Gaussian. Specifically, the covariance matrix of the feature weights transforms from an isotropic form to:
$\Gamma = A I_d + B w^* (w^*)^\top$
where $A$ and $B$ are constants dependent on the step size, data dimensions, and the link function $g$ . This leads to a data-adaptive kernel $k_1$ that can be expressed as:
$k_1(x, x') = k_0(\Gamma^{1/2}x, \Gamma^{1/2}x')$
This formulation interprets feature learning as a distributional transformation in either the parameter space (shifting the weight distribution) or the input space (transforming the data representation).

B. Spectral Expansion and Kernel Deformation

The authors perform a Taylor expansion of the new kernel $k_1$ around the initial isotropic kernel $k_0$ . They demonstrate that the "spike" (the rank-one update aligned with $w^*$ ) introduces higher-order terms that vanish as the dimension grows, leaving a dominant first-order perturbation.

Result: The feature learning process effectively imposes an additional, target-dependent kernel component.
Mechanism: This component selectively amplifies eigenvalues aligned with the target direction $w^*$ and mixes leading eigenfunctions.

C. Explicit Characterization for ReLU Activations

For the specific case of ReLU activation, the authors provide a precise spectral analysis:

Linear Eigenspace: The linear function aligned with the target, $\psi^*(x) = \langle x, w^* \rangle$ , receives a selective boost to its eigenvalue. Linear functions orthogonal to $w^*$ remain unaffected.
Top Eigenspace Mixing: The top eigenfunction of the updated operator is not purely the constant harmonic (as in the initial kernel) but becomes a superposition of the constant radial mode ( $Y_0$ ) and a target-aligned quadratic harmonic ( $Y_2$ ).
Coupling: This reveals that feature learning couples the top radial mode with a quadratic harmonic aligned with the signal, effectively reshaping the geometry of the function space to prioritize directions containing the signal.

D. Numerical Validation

Experiments on synthetic data confirm the theoretical findings:

Alignment: The leading eigenvector of the updated kernel ( $k_1$ ) shows increasing alignment with the quadratic feature $Y_2$ as the spike strength $B$ increases, whereas the initial kernel ( $k_0$ ) shows no such alignment.
Generalization: Neural networks trained with a single gradient step converge toward the performance of the spiked kernel $k_1$ , outperforming the fixed kernel $k_0$ in learning the target function, particularly when the target contains non-linear components.

4. Significance and Claims

The paper provides a precise function-space perspective on early-stage feature learning. Its primary claim is that gradient descent does not merely rescale a fixed kernel; rather, it induces a data-adaptive deformation of the function space.

Mechanism of Learning: Feature learning is characterized as a transformation that preferentially enhances directions in the function space aligned with the signal in the data.
Step Size Dependence: The magnitude of this transformation is modulated by the step size; larger steps induce stronger deformations and increase the contribution of both linear and higher-order non-linear features.
Theoretical Bridge: The results bridge the gap between data-adaptive kernels and neural networks, suggesting that the "magic" of feature learning can be understood through the lens of how the kernel's spectral structure is modified to align with the target function.

The authors conclude that understanding these distributional shifts offers a potential pathway to "skip" early training phases by directly initializing networks with these adapted kernels, though they note this as a direction for future research rather than a demonstrated application in the current work.