🤖 machine learning

Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks

This paper investigates how width-dependent hyperparameters and $\ell_2$ -regularization shape the loss landscape of two-layer ReLU networks, deriving conditions for zero-solution collapse, providing an analytical solution for one-dimensional inputs, and revealing that AdamW prevents parameter collapse while SGD does not.

Original authors: Haruka Eshima, Makoto Yamada

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haruka Eshima, Makoto Yamada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize cats. You give it a brain made of layers of tiny switches, and you let it practice by showing it thousands of pictures. But here's the catch: the robot's brain is a bit messy. Sometimes, when you try to teach it, the whole thing just shuts down and forgets everything, ending up with a brain full of zeros. This is a bit like trying to balance a house of cards on a windy day; if the wind (or the math behind the training) is too strong, the whole structure collapses.

In the world of machine learning, scientists study the "loss landscape." Think of this as a giant, bumpy mountain range where the height of the terrain represents how bad the robot is at its job. The goal is to find the deepest valley, which means the robot is doing its best. But this landscape is tricky. It changes shape depending on how wide the robot's brain is (how many switches it has) and how much "weight decay" is applied. Weight decay is like a gentle hand constantly pushing the robot's brain toward simplicity, trying to stop it from getting too complicated. The big question is: does this gentle push help the robot learn, or does it accidentally push the whole brain into a flat, useless zero?

This paper, titled "Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networks," dives deep into this exact problem. The authors, Haruka Eshima and Makoto Yamada, looked at a specific type of simple neural network (a two-layer network with ReLU activation) to see what happens when you change the size of the network and the strength of the weight decay. They wanted to know: under what conditions does the robot's brain collapse into nothingness, and can we stop it?

The researchers found that the answer depends heavily on how you scale things up. They discovered a mathematical "tipping point." If you make the network wider (add more switches) but keep the weight decay too strong relative to that width, the robot's brain will inevitably collapse. The only way to reach the bottom of the valley is to have every single switch turned off, meaning the robot learns nothing. It's like trying to fill a swimming pool with a tiny cup while the drain is wide open; no matter how much you pour, the pool stays empty.

However, the paper also reveals a fascinating twist involving how the robot learns. The authors ran computer simulations to see what happens when you use different "optimizers"—the algorithms that guide the robot's learning steps. They found that if you use a standard method called SGD (Stochastic Gradient Descent), the robot's brain does indeed collapse into zero when the weight decay is too strong, just as the math predicted. But, if you use a different method called AdamW, the robot survives! Even when the math says it should collapse, AdamW keeps the brain active and learning. It's as if AdamW has a special trick, like a safety harness, that prevents the robot from falling into the zero-hole, even when the terrain looks dangerous.

The team also zoomed in on a very specific, simplified scenario where the robot only sees one number at a time (a one-dimensional input). In this controlled setting, they were able to write down an exact map of the best possible solutions. They found that adding weight decay acts like a sculptor's chisel. Without it, the best solutions are scattered all over the place, forming a huge, connected, and unbounded cloud of possibilities. But with weight decay, the cloud shrinks and becomes much more organized. The "connectivity"—how easy it is to walk from one good solution to another without falling off a cliff—stays the same regardless of how wide the network is. However, the "dimensionality"—the number of ways the robot can wiggle its brain to find a solution—shrinks significantly as the network gets wider. It's like taking a giant, sprawling playground and turning it into a narrow, well-defined track.

In short, this paper shows that the size of your neural network and the strength of your regularization are a delicate dance. If you don't match them correctly, the network collapses. But if you choose the right learning algorithm (like AdamW), you can keep the dance going even when the conditions seem impossible. The results are based on rigorous mathematical proofs for the collapse and the simplified one-dimensional case, and supported by numerical experiments on real-world data like yacht hydrodynamics and MNIST digits. While the math proves the collapse happens under certain conditions, the survival of the network with AdamW is something the authors observed in their simulations, suggesting that the choice of optimizer is a critical, perhaps hidden, factor in why deep learning works so well in practice.

Technical Summary: Effects of Width-Dependent Model Hyperparameters and $\ell_2$ -Regularization on the Loss Landscape of Two-Layer ReLU Networks

Problem Statement
The theoretical properties of deep neural networks, particularly two-layer ReLU networks, remain poorly understood when explicit weight decay ( $\ell_2$ -regularization) is applied. While recent works have characterized global optima and their geometric structures for regularized networks, the systematic effects of width-dependent hyperparameter scalings on these loss landscapes have not been fully explored. Specifically, it is unclear how scaling the network width ( $m$ ) relative to the scaling of the output ( $\alpha$ ), initialization ( $\tau$ ), and the regularization coefficient ( $\beta$ ) influences the existence, dimensionality, and connectivity of global minima.

Methodology
The authors analyze a two-layer ReLU network $f_\theta: \mathbb{R}^d \to \mathbb{R}$ defined as:
$f_\theta(x) = \frac{1}{\alpha} \sum_{j=1}^m (x u_j)_+ \omega_j$
where $\theta = (u_j, \omega_j)_{j=1}^m$ . The study focuses on minimizing a loss function $\mathcal{L}_\ell(\theta) = \ell(\theta) + \frac{\beta}{2} \sum_{j=1}^m (\|u_j\|_2^2 + \omega_j^2)$ .

The core methodological approach involves introducing a width-dependent hyperparameter setting where parameters scale with the network width $m$ :

Output scaling: $\alpha = m^a$
Initialization scales: $\tau_1 = m^{-b_1}, \tau_2 = m^{-b_2}$
Regularization coefficient: $\beta = m^{-\delta}$

The analysis proceeds in two main directions:

General Dimensionality ( $d$ ): Deriving sufficient conditions under which the global minima collapse to the zero solution based on the relationship between the scaling exponents $a$ and $\delta$ .
One-Dimensional Input ( $d=1$ ): Restricting the input dimension to one to derive analytical closed-form solutions for the globally optimal parameter sets. This allows for a precise characterization of the geometry (dimension, boundedness, and connectivity) of the solution sets for both unregularized ( $\beta=0$ ) and regularized ( $\beta>0$ ) cases. The proofs leverage the convex formulation of two-layer ReLU networks introduced by Pilanci and Ergen [31], which becomes tractable in the 1D setting.

Key Contributions and Results

Collapse of Global Optima:
The paper establishes a sufficient condition for the collapse of global minima to the zero solution. Theorem 1 proves that if the regularization coefficient scales faster than the network output scaling (i.e., $\delta < a$ ), then for sufficiently large $m$ , the unique global minimum is the zero weight vector. This holds for most convex loss functions used in practice.
Optimizer-Dependent Collapse:
Numerical experiments reveal a critical discrepancy between theoretical global minima and practical training dynamics. When $\delta < a$ (conditions favoring collapse), SGD causes the learned parameters to collapse to zero, consistent with Theorem 1. However, AdamW (and Adam) prevents this collapse, maintaining non-zero parameters. The authors suggest this may be related to AdamW's decoupled weight decay and its implicit bias toward $\ell_\infty$ -constrained optimization, which differs from optimizing the explicit $\ell_2$ -regularized objective.
Analytical Characterization of 1D Global Minima:
For $d=1$ , the authors derive exact descriptions of the solution sets:
- Unregularized ( $\beta=0$ ): The set of global minima $\phi^*(m)$ $ϕ^{*} (m)$ is characterized by constraints on the sums of products $u_j \omega_j$ $u_{j} ω_{j}$ .
  - Connectivity: For $m=1$ , the set is empty; for $m=2$ , it has exactly 2 connected components; for $m \ge 3$ , it is connected.
  - Dimension: The dimension is $2m-2$ , and the set is unbounded.
- Regularized ( $\beta>0$ ): The set of global minima $\Theta^*(m)$ $Θ^{*} (m)$ is significantly more constrained.
  - Connectivity: The paper identifies critical widths $M$ and $\bar{M}$ (dependent on hyperparameters) that dictate phase transitions. For $m < M$ , the solution is the zero vector. For $m > M$ , the set becomes connected. Unlike the unregularized case, the connectivity behavior is more restricted, with only three possible behaviors (singleton, finite set, or connected) for non-empty sets.
  - Dimension: The dimension of the solution set is reduced to $m-2$ .
  - Boundedness: Unlike the unregularized case, the regularized solution set is bounded. The norm of the optimal parameters scales as $\Theta(m^{a/2})$ .
Width-Dependent Effects:
The analysis shows that $\ell_2$ -regularization has a width-invariant effect on connectivity (once the network is wide enough to be non-trivial) but a dimensionality-reducing effect that strengthens with width. Specifically, the difference in dimension between the parameter space ( $2m$ ) and the regularized solution set grows linearly with $m$ (from a difference of 2 in the unregularized case to a difference of $m+2$ in the regularized case).

Significance
The paper provides a rigorous theoretical framework for understanding how hyperparameter scaling interacts with regularization in overparameterized networks. It highlights that the geometry of the loss landscape is not static but depends critically on the relative scaling of the regularization coefficient and the network width. The findings offer a theoretical explanation for the empirical success of AdamW in preventing parameter collapse in regimes where SGD fails, and they clarify how regularization fundamentally alters the geometry of optimal solutions by reducing dimensionality and enforcing boundedness, effects that become more pronounced as the network width increases.

More like this