🤖 machine learning

A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability

This paper establishes a bifurcation theory framework for gradient descent on the Edge of Stability in overparameterized neural networks, demonstrating that stable training arises from a flip bifurcation in the normal direction and tangent drift toward lower sharpness, thereby proving convergence to the minimizing manifold and unifying prior stability conditions.

Original authors: Eric Gan

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eric Gan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to walk down a steep, winding mountain path to reach a valley floor (the "minimum loss" where your AI model works best).

In the old rules of physics (classical math), if you take steps that are too big, you will overshoot the path, bounce off the side, and fall into a ravine. To stay safe, you must take tiny, careful steps.

But here is the mystery that modern AI researchers face: AI models are routinely taking huge, reckless steps. In fact, they are stepping so hard that they are technically "unstable" according to the old rules. Yet, instead of falling off a cliff, they somehow stay on the path, wobble a bit, and eventually reach the valley. This strange phenomenon is called the Edge of Stability (EoS).

This paper by Eric Gan provides a new map to explain how this magic happens, specifically for complex AI models that have millions of parameters (overparameterized networks).

Here is the breakdown of the paper's theory using simple analogies:

1. The Landscape: A Valley with a Flat Floor

In simple math problems, the bottom of the valley is a single point. But in modern AI, the "bottom" isn't a point; it's a long, flat valley floor (a "manifold of minimizers").

The Normal Direction: Imagine you are standing on a tightrope stretched across this valley. If you step too far left or right (the "normal" direction), you might fall. This is where the "instability" happens.
The Tangent Direction: Imagine walking along the tightrope. You can move forward or backward without falling off the side. This is the "tangent" direction.

2. The Wobble: The "Flip" (Normal Direction)

When the AI takes a step that is too big (overshooting the stability threshold), it doesn't crash. Instead, it starts to oscillate.

The Analogy: Think of a pendulum. If you push it just right, it swings back and forth in a perfect rhythm. The paper explains that when the AI is on the "Edge of Stability," it enters a state where it swings back and forth (a "period-2 oscillation") in the normal direction.
The Secret Sauce: The paper uses a concept called Bifurcation Theory (the study of how systems change behavior) to show that this wobble is actually stable. It's like a tightrope walker who leans left, then right, then left again, finding a rhythm that keeps them from falling.
The "Lyapunov Coefficient": This is a fancy math number that acts like a stability meter. The paper proves that as long as this number is positive, the "wobble" will keep the AI from flying off the path.

3. The Drift: Moving Toward Safety (Tangent Direction)

While the AI is busy wobbly-swinging side-to-side (normal direction), something else is happening. It is slowly drifting along the path (tangent direction).

The Analogy: Imagine the tightrope is slightly tilted. Even though you are swinging side-to-side, gravity is slowly pulling you toward the part of the rope that is flatter and safer.
The Result: The AI naturally drifts toward areas where the "sharpness" (the steepness of the hill) is lower. As it drifts into these flatter areas, the wobble becomes less dangerous, and the AI finally settles down and converges to the solution.

4. Connecting the Dots

Previous research tried to explain this using very simple, 2D examples (like a ball rolling on a flat sheet). This paper says, "We can explain this for real AI models too."

It shows that the old, simple explanations are just special cases of this new, bigger theory.
It proves that even with the massive complexity of modern neural networks, the same rules apply: The AI stabilizes itself by wobbly-oscillating in one direction while slowly drifting to a safer spot in the other.

Summary

The paper argues that the "Edge of Stability" isn't a bug; it's a feature.

The Wobble: The AI oscillates back and forth in a stable rhythm (governed by a "Lyapunov coefficient") instead of crashing.
The Drift: While wobbling, it slowly slides toward flatter, safer ground.
The Outcome: The combination of the stable wobble and the slow drift allows the AI to train successfully even when taking steps that should theoretically be too big.

In short: The AI doesn't fall off the cliff because it learns to dance on the edge, using the very instability that should have destroyed it to find its way to the bottom.

Technical Summary: A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability

Problem Statement
The "Edge of Stability" (EoS) is a ubiquitous phenomenon in modern deep learning where gradient descent (GD) operates with a learning rate $\eta$ and loss sharpness $\lambda$ (the largest eigenvalue of the Hessian) such that $\eta\lambda > 2$ . Classical convergence theory dictates that GD diverges when $\eta\lambda \geq 2$ ; however, in practice, models trained under these conditions often exhibit non-monotonic loss evolution yet converge reliably. While prior rigorous analyses have explained EoS in minimalist, low-dimensional settings (e.g., scalar losses or specific structured functions), they fail to capture the geometry of realistic, overparameterized neural networks. Specifically, existing frameworks do not adequately address the presence of a continuous manifold of global minimizers, a characteristic feature of overparameterized systems where the Hessian is rank-deficient.

Methodology
The paper develops a bifurcation theory framework tailored to the geometry of overparameterized neural networks. The core methodological innovation is the decomposition of training dynamics into two orthogonal components relative to the manifold of minimizers $\mathcal{M}$ :

Normal Dynamics: The behavior perpendicular to the manifold of minimizers.
Tangent Dynamics: The behavior along the manifold of minimizers.

The authors model the GD update as a discrete dynamical system $x_{t+1} = f(x_t)$ . At the EoS threshold ( $\eta\lambda_{\max} = 2$ ), the Jacobian of the system possesses a critical eigenvalue $\lambda = -1$ . The authors apply bifurcation theory, specifically analyzing flip bifurcations (period-doubling bifurcations), to characterize the stability of the system.

Key steps in the methodology include:

Center Manifold Reduction: Utilizing the Center Manifold Theorem to reduce the high-dimensional dynamics to a lower-dimensional invariant manifold near the critical point.
Lyapunov Coefficient Analysis: Calculating the first Lyapunov coefficient ( $c_1$ ) to determine the stability of the bifurcation. A positive $c_1$ indicates a supercritical flip bifurcation, leading to stable period-2 oscillations rather than divergence.
Decoupling: Deriving explicit dynamical equations for the normal and tangent components. The normal component is shown to stabilize via period-2 oscillations governed by $c_1$ , while the tangent component exhibits a drift toward regions of decreasing sharpness.

Key Contributions

General Framework for Overparameterized Networks: The paper extends EoS analysis beyond scalar or low-dimensional losses to general overparameterized networks by explicitly accounting for the manifold of minimizers.
Decomposition of Dynamics: The authors provide a rigorous decomposition of EoS dynamics, showing that convergence arises from the interplay between stable normal oscillations and tangent drift.
Convergence Proof at the Threshold: Under mild spectral and geometric assumptions (specifically a spectral gap and the Morse-Bott condition), the paper proves that gradient descent converges to the minimizing manifold when initialized near the EoS threshold ( $\eta = 2/\lambda_{\max}$ ).
Unification of Prior Work: The framework demonstrates that the "product-stability" condition proposed by Gan (2026) for losses of the form $L(x,y) = f(xy)$ is a specific instance of the general bifurcation framework. The authors prove that the product-stability condition is directly related to the sign of the first Lyapunov coefficient in this specific setting.

Results

Stability Mechanism: The paper establishes that stable training at the EoS threshold is governed by the sign of the first Lyapunov coefficient ( $c_1$ ). When $c_1 > 0$ , the system undergoes a supercritical flip bifurcation, creating a stable period-2 orbit in the normal direction that prevents divergence.
Drift Dynamics: In the tangent direction, the iterates drift along the manifold of minimizers in the direction of decreasing sharpness. This drift ensures that the system eventually moves to regions where the sharpness drops below the EoS threshold, facilitating convergence to a specific point on the manifold.
Theorem 4.4: The main theoretical result states that if $c_1(x^*) > 0$ and the gradient of sharpness along the tangent space is non-zero, gradient descent with $\eta = 2/\lambda_{\max}(x^*)$ converges to a point on the manifold $\mathcal{M}$ .
Illustrative Example: The paper validates the framework by applying it to the loss $L(x,y) = f(xy)$. It shows that the sign of the first Lyapunov coefficient for $L$ aligns with the sign of the product-stability parameter $\alpha_f$ defined in Gan (2026), thereby unifying the scalar analysis with the general manifold framework.

Significance and Claims
The paper claims to provide the first rigorous theoretical framework that explains stable gradient descent convergence on the edge of stability specifically for overparameterized neural networks with manifolds of minimizers. By shifting the analysis from classical descent lemmas (which require $\eta < 2/\lambda_{\max}$ ) to bifurcation theory, the work explains how convergence is possible precisely at the threshold $\eta = 2/\lambda_{\max}$ .

The authors position their work as a unification of prior minimalist analyses (such as those by Gan, 2026; Zhu et al., 2023; and Wang et al., 2023) under a general geometric and dynamical systems perspective. They note that while empirical evidence suggests practical models satisfy the condition $c_1 > 0$ , the theoretical explanation for why this holds from first principles remains an open question. Similarly, the mechanism driving the initial "progressive sharpening" that leads training iterates into the EoS regime is identified as an unresolved aspect of the phenomenon.