🤖 AI

Flat Channels to Infinity in Neural Loss Landscapes

This paper identifies and characterizes "channels to infinity" in neural network loss landscapes where optimization algorithms converge to quasi-flat regions with diverging weights that asymptotically implement gated linear units, revealing a surprising computational capability of fully connected layers.

Original authors: Flavio Martinelli, Alexander Van Meegen, Berfin Şimşek, Wulfram Gerstner, Johanni Brea

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Flavio Martinelli, Alexander Van Meegen, Berfin Şimşek, Wulfram Gerstner, Johanni Brea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiking in a vast, foggy mountain range. This landscape represents the "loss landscape" of a neural network—a map where every point is a different setting for the computer's brain, and the height of the terrain represents how badly the computer is making mistakes. Usually, hikers (optimization algorithms) want to find the deepest valley (the lowest error) to stop.

This paper discovers a strange, hidden feature in this mountain range: infinite, flat tunnels that lead to the edge of the world.

Here is the breakdown of what the authors found, using simple analogies:

1. The "Ghost" Saddle Lines

First, the authors looked at a known phenomenon. If you take a working neural network and copy one of its neurons (like cloning a worker in a factory), you create a "saddle line."

The Analogy: Imagine a mountain ridge. If you stand on this ridge, you are at a specific height. If you walk along the ridge, the height doesn't change. In math terms, this is a line of "critical points" where the computer isn't getting better or worse.
The Twist: The authors found that next to these ridges, there are these weird, flat tunnels.

2. The "Channels to Infinity"

These are the paper's main discovery. They are paths in the landscape that look like flat valleys but actually stretch on forever.

What happens inside: As the computer travels down this tunnel, two things happen simultaneously:
1. The "Input" weights (the sensors): Two neurons start to look exactly the same. Their settings become identical.
2. The "Output" weights (the volume knobs): The volume knobs for these two neurons get turned up to infinity. One goes to positive infinity, the other to negative infinity.
The Paradox: Even though the knobs are spinning wildly to infinity, the computer's actual output remains stable and the error (the height of the mountain) stays incredibly low. It's like a car driving down a road where the engine is revving to 10,000 RPM, but the car is moving at a constant, smooth speed.

3. Why the Hikers Get Stuck

The authors ran simulations using standard hiking tools (optimization algorithms like SGD and ADAM).

The Trap: These tools are great at finding the bottom of a valley, but they are terrible at realizing they are on a path that goes on forever.
The Illusion: Because the path is so flat, the hikers think they have reached the bottom of a local valley (a "local minimum"). They stop, thinking they are done.
The Reality: They are actually just standing in a very flat, very long tunnel that leads to infinity. If you could keep going, the error would keep dropping, ever so slightly, forever.

4. The Magic Trick: The "Gated Linear Unit"

So, what is the computer actually doing at the end of this infinite tunnel? It's not broken; it's performing a clever mathematical magic trick.

The Analogy: Imagine you have two identical twins (the neurons) standing next to each other. One is shouting a message very loudly, and the other is whispering the exact opposite message very loudly.
The Result: When you add their voices together, the loud parts cancel out, but because they are slightly different in position, a new, very specific sound emerges.
The Math: The authors show that as the twins get closer together and their voices get louder, the combination creates a "Gated Linear Unit."
- Think of this as a smart switch. It takes a standard signal and multiplies it by a "gate" (a filter).
- This allows the network to perform a new type of calculation it couldn't do easily before: it can calculate the rate of change (the derivative) of a function. It's like the network suddenly learned how to take a snapshot of how fast things are changing, not just what they are.

5. Why This Matters (According to the Paper)

The paper suggests that these "infinite tunnels" are actually a good thing, not a bug.

Benign Nature: Even though the math looks scary (infinite numbers!), the landscape is "benign" (safe). The computer doesn't crash; it just finds a very efficient way to solve the problem.
Flatness: These tunnels are incredibly flat. In the world of neural networks, flat areas are often associated with better generalization (the ability to handle new, unseen data).
The "Edge of Stability": The paper notes that standard computers (using standard step sizes) get stuck at the "edge" of these tunnels. They can't go all the way to infinity because their "step size" is too big for the increasingly sharp turns of the tunnel, but they do get close enough to unlock that new "Gated Linear Unit" capability.

Summary

The paper reveals that neural networks have secret, infinite highways running parallel to known mountain ridges. When a computer drives down these highways, it doesn't crash; instead, it transforms two neurons into a powerful, specialized tool (a gated linear unit) that helps it solve complex problems. Standard training methods often mistake these infinite highways for dead-end valleys, but the destination is actually a place of new computational power.

Technical Summary: Flat Channels to Infinity in Neural Loss Landscapes

Problem Statement
Neural network loss landscapes are inherently non-convex, yet gradient-based optimization methods consistently find good solutions. While the benign nature of saddle points induced by neuron duplication (symmetry-induced lines of critical points) is theoretically understood, the behavior of optimization dynamics in the vicinity of these structures remains partially unexplored. Specifically, it is unclear whether gradient flow converges to finite local minima or if there exist unbounded regions in the parameter space that attract optimization trajectories. This paper investigates the existence and nature of "channels to infinity"—regions where the loss decreases extremely slowly while parameters diverge.

Methodology
The authors analyze fully connected neural networks with a single hidden layer using mean-squared error (MSE) loss and smooth activation functions (e.g., softplus, erf). The study employs a combination of theoretical analysis and empirical simulation:

Neuron Duplication and Saddle Lines: Building on the work of Fukumizu and Amari [1], the authors utilize neuron duplication to generate lines of critical points (saddle lines) in the loss landscape of a network with $r+1$ neurons derived from a network with $r$ neurons.
Gradient Flow Integration: The authors use ODE solvers to integrate gradient flow dynamics ( $\dot{\theta} = -\nabla_\theta L$ ) from random initializations (Glorot normal) and from perturbed saddle lines.
Perturbation Analysis: To characterize the stability of saddle lines, they perturb solutions along the eigenvector corresponding to the smallest (most negative) eigenvalue of the loss Hessian.
Reparameterization: To understand the functional limit of these channels, the authors reparameterize pairs of neurons $(i, j)$ with input weights $w_i, w_j$ and output weights $a_i, a_j$ into a central difference form involving a parameter $\epsilon = \|w_i - w_j\|/2$ .
Jump Procedure: To traverse channels where convergence is extremely slow, they implement a "jump procedure" that artificially reduces $\epsilon$ while maintaining other parameters, allowing them to observe the asymptotic behavior of the loss and parameters.
Toy Models: A specific 1D target function approximation using two neurons without biases is analyzed under population loss (infinite data) to rigorously prove the stability of minima at infinity.

Key Contributions and Results

Identification of Channels to Infinity: The authors identify a novel structure in the loss landscape: channels running asymptotically parallel to symmetry-induced saddle lines. Within these channels:
- The output weights of at least two neurons ( $a_i, a_j$ ) diverge to $\pm \infty$ with opposite signs ( $a_i a_j < 0$ ).
- The input weight vectors ( $w_i, w_j$ ) converge to the same finite value.
- The loss decreases extremely slowly (asymptotically flat) along the channel direction.
Empirical Prevalence: In diverse regression settings (including modified Rosenbrock functions and Gaussian Processes with varying kernel scales), gradient-based optimizers (including SGD and ADAM) reach these channels with high probability from standard random initializations. The probability of finding such channels increases with the roughness of the target function.
Functional Interpretation (Gated Linear Units): Through Taylor expansion of the neuron pair output as $\epsilon \to 0$ , the authors demonstrate that the limit of these channels implements a gated linear unit (GLU). Specifically:
$a_i\sigma(w_i \cdot x) + a_j\sigma(w_j \cdot x) \to c\sigma(w \cdot x) + (v \cdot x)\sigma'(w \cdot x)$
where the second term represents a linear transformation gated by the derivative of the activation function. For softplus activation, the gate is the standard sigmoid function.
Stability and Geometry:
- Stability: In the limit $\epsilon \to 0$ , the loss scales as $L \approx L_0 + \epsilon^2 h(\theta_0)$ . If $h(\theta_0) > 0$ , the channel leads to a stable local minimum at infinity. The authors prove this stability for specific toy examples.
- Sharpness vs. Flatness: While the channel is extremely flat along the direction of divergence (loss decreases as $\epsilon^3$ with respect to parameter updates), the Hessian's maximum eigenvalue (sharpness) diverges as $1/\epsilon^2$ . This creates a scenario where optimizers get stuck at the "edge of stability" [47], appearing as flat local minima with finite parameters in practice, even though the true minimum lies at infinity.
Multi-layer and Multi-neuron Channels: The phenomenon is not limited to single hidden layers or pairs of neurons. Channels can appear in any layer of deep MLPs, involve multiple neurons (leading to higher-order derivative approximations), and form multi-dimensional flat regions.

Significance and Claims
The paper claims to provide a comprehensive picture of "quasi-flat" regions in neural loss landscapes that are often mistaken for finite local minima. The emergence of gated linear units at the end of these channels suggests that fully connected layers possess computational capabilities (approximating derivatives via gating) that are naturally accessed by gradient flow, even if the parameters diverge.

The authors argue that these channels contribute to the "benign" nature of non-convex optimization landscapes. Although the parameters diverge, the functional output converges to a useful representation (GLU). The paper notes that in practical training with finite step sizes or regularization, optimizers often halt within the channel (at the edge of stability), effectively implementing the GLU approximation without reaching infinite parameter norms. This phenomenon offers a new perspective on generalization, model fusion, and the intrinsic capabilities of neural network architectures, suggesting that the "flatness" observed in training may sometimes be a manifestation of these asymptotic structures rather than simple finite minima.

The work is framed as a characterization of the loss landscape geometry and gradient dynamics, highlighting that the "minima at infinity" found here differ from those in separable classification problems and represent a generic mechanism in regression settings with smooth activations.