📊 statistics

Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization

This paper argues that understanding the generalization of overparameterized networks requires analyzing the dynamic trade-off between norm minimization and sharpness reduction, demonstrating that learning rates interpolate between these biases and that neither alone suffices to minimize generalization error.

Original authors: Maria Matveev, Vit Fojtik, Hung-Hsu Chou, Gitta Kutyniok, Johannes Maly

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maria Matveev, Vit Fojtik, Hung-Hsu Chou, Gitta Kutyniok, Johannes Maly

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Goldilocks Problem of AI Training

Imagine you are teaching a student (a neural network) to solve a puzzle. You want the student to learn the rules so well that they can solve new puzzles they've never seen before (this is called generalization).

For a long time, researchers thought there was one "magic trick" that made students good at this. They believed the training process (called Gradient Descent) naturally pushed the student toward the simplest, most "compact" solution. In math terms, this is called minimizing the Norm (keeping the numbers in the model small and tidy).

However, recent observations showed something else happening. When you train with a "fast" learning rate, the student seems to naturally avoid "spiky" or unstable solutions and instead finds "flat," smooth solutions. In math terms, this is called minimizing Sharpness.

The paper's main discovery: These two "magic tricks" (keeping things small vs. keeping things flat) are actually fighting each other. You can't have both at the same time. The paper argues that the secret to a good AI isn't just one of these traits, but finding the perfect balance between them, which is controlled by how fast you teach the student (the Learning Rate).

The Two Competing Biases

To understand the conflict, let's use two metaphors:

1. The "Tiny Backpack" Bias (Norm Minimization)

What it is: When you teach very slowly (tiny learning rate), the algorithm acts like a hiker trying to carry the lightest possible backpack. It tries to keep the "weight" of the model (the size of the numbers inside) as small as possible.
The result: You get a very simple, compact model.
The paper's finding: While simple is usually good, the paper shows that the simplest possible model isn't always the one that performs best on new data.

2. The "Flat Valley" Bias (Sharpness Minimization)

What it is: When you teach at a moderate-to-fast speed, the algorithm acts like a hiker looking for a wide, flat valley floor rather than a narrow, precarious peak. If the solution is "sharp" (like standing on a needle), a tiny mistake in the next step will send you tumbling. If it's "flat" (like a wide plateau), you can wobble a bit and still stay safe.
The result: You get a model that is very stable and robust to small changes.
The paper's finding: While stability is great, the flattest possible solution isn't always the best either.

The "Edge of Stability" Dance

The paper introduces a fascinating concept called the Edge of Stability (EoS).

Imagine you are walking a tightrope.

Too slow (Small Learning Rate): You walk very carefully, staying perfectly in the middle. You end up with a "Tiny Backpack" (low norm), but you might be walking on a narrow path that isn't the best route.
Too fast (Large Learning Rate): You run so fast you start bouncing off the tightrope. You fall off (the training crashes).
Just right (The Edge of Stability): You run at a speed where you almost lose your balance. You wobble back and forth, but you don't fall. In this state, the algorithm naturally pushes you toward the "Flat Valley" (low sharpness).

The Conflict: The paper shows that as you speed up your walking (increase the learning rate) to get to this "wobbly but stable" state, your backpack gets heavier and heavier (the Norm increases).

Slow speed: Light backpack, narrow path.
Fast speed: Heavy backpack, wide, flat valley.

You cannot have a light backpack and a wide valley simultaneously. They are trade-offs.

The "U-Shaped" Secret

The most important finding is about Generalization (how well the model works on new data).

If you plot the performance of the model against the learning speed, you often get a U-shape:

Too Slow: The model is too simple (low norm) and misses the nuance of the data. Performance is okay, but not great.
Too Fast: The model is too chaotic (high norm, even if flat) and overfits or becomes unstable. Performance drops.
Just Right (The Middle): The sweet spot is in the middle. Here, the model has a "medium-sized backpack" and is "moderately flat."

The Analogy: Think of tuning a radio.

Turn the dial too far left (too slow), and you hear static (underfitting).
Turn it too far right (too fast), and you hear static again (instability).
The clearest signal is found in the middle, where you have balanced the two opposing forces.

The Theoretical Proof (The Simple Model)

To prove this isn't just a fluke of complex computers, the authors built a tiny, simple math model (a "toy" network).

They showed that the solution with the smallest backpack (lowest norm) is in a completely different location than the solution with the flattest valley (lowest sharpness).
They proved that the best solution (the one that predicts new data best) is often a third option: a spot right in between the two extremes.

Conclusion: It's a Balancing Act

The paper concludes that we cannot explain why AI works so well by looking at just one factor (like "it finds the simplest solution").

Instead, the learning rate acts like a dimmer switch that balances two conflicting desires:

Keeping the model small and simple.
Keeping the model stable and flat.

The "magic" of deep learning happens when we tune this switch to find the perfect compromise between these two opposing forces. The paper argues that focusing on only one of these biases is insufficient; we must understand the dynamic trade-off between them.

Technical Summary: Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization

Problem Statement

The remarkable generalization capabilities of over-parameterized neural networks are frequently attributed to the implicit biases of optimization algorithms like Gradient Descent (GD). Two dominant theories have emerged to explain this phenomenon:

Norm Minimization: In the limit of vanishing learning rates (or Gradient Flow), GD exhibits an implicit bias toward solutions with minimal parameter norms (e.g., $\ell_1$ or $\ell_2$ norms), particularly in linear and diagonal network settings.
Sharpness Minimization: In the "Edge of Stability" (EoS) regime, where learning rates are finite and large enough to cause non-monotonic loss behavior, GD is observed to converge toward solutions with low sharpness (specifically, the spectral norm of the Hessian is bounded by $2/\eta$ ).

The central problem addressed by this work is that these two forms of implicit regularization are often analyzed in isolation. It remains unclear how they interact when GD is trained with the finite, non-negligible learning rates used in practice. Specifically, does a single bias (either norm or sharpness) suffice to explain generalization, or is there a dynamic trade-off between them?

Methodology

The authors employ a dual approach combining extensive empirical analysis across diverse settings and a rigorous theoretical analysis on a simplified model.

Empirical Analysis

The authors conduct systematic experiments across a wide range of configurations to isolate the effects of the learning rate $\eta$ :

Datasets: MNIST and CIFAR-10 (both subsets of 5k and full sets).
Architectures: Fully Connected Networks (FCNs) with varying widths and depths, Convolutional Neural Networks (CNNs), ResNets, and Vision Transformers (ViT).
Hyperparameters: A systematic sweep of learning rates, including a fine-grained grid around the critical transition point and a coarse grid for larger rates.
Metrics: The study tracks the final parameter norm ( $\ell_1, \ell_2$ , nuclear norm), the sharpness of the training loss (operator norm of the Hessian), and the test error.
Control: Experiments use full-batch GD to eliminate stochasticity, fixed initializations to ensure comparability, and various loss functions (MSE, Cross-Entropy) and activation functions (ReLU, tanh).

Theoretical Analysis

To provide mechanistic insight, the authors analyze a simplified regression task: training a shallow diagonal linear network with shared weights on a single data point using square loss.

They explicitly characterize the solution manifold where the training loss is zero.
They derive the specific solutions that minimize the parameter norm versus those that minimize sharpness.
They analyze the generalization error (risk) of these distinct solutions and compare them to the actual convergence behavior of GD under varying learning rates.

Key Contributions

1. Discovery of a Learning Rate-Dependent Trade-off

The paper empirically demonstrates that the learning rate acts as a control knob interpolating between two distinct implicit biases:

Flow-Aligned Regime ( $\eta < \eta_c$ ): When the learning rate is below a critical threshold $\eta_c \approx 2/s_{GF}$ (where $s_{GF}$ is the maximum sharpness of the gradient flow trajectory), GD behaves similarly to Gradient Flow. Both the parameter norm and sharpness remain relatively constant.
Edge-of-Stability Regime ( $\eta > \eta_c$ ): As the learning rate exceeds $\eta_c$ , the system enters the EoS regime. Here, a sharp phase transition occurs: the final sharpness of the model decreases hyperbolically (tracking $2/\eta$ ), while the parameter norm increases approximately linearly.
Conclusion: There is an inherent trade-off; one cannot simultaneously minimize both norm and sharpness by simply tuning the learning rate.

2. Generalization is Not Monotonic with Single Biases

The authors show that the "best" generalization performance does not necessarily align with either extreme of the learning rate spectrum:

Low Learning Rates: Solutions with minimal norm (flow-aligned) often do not yield the lowest test error.
High Learning Rates: Solutions with minimal sharpness (EoS) can also lead to suboptimal generalization if the norm becomes too large.
U-Shaped Curves: In many settings, the test error follows a U-shaped curve with respect to the learning rate. The optimal generalization is achieved at intermediate learning rates where the biases toward low norm and low sharpness are balanced.

3. Theoretical Counterexample

Using the diagonal linear network model, the authors prove that:

The set of parameters minimizing the norm and the set of parameters minimizing sharpness are geometrically distinct and often disjoint.
Neither the norm-minimizer nor the sharpness-minimizer necessarily achieves the lowest expected generalization error.
The learning rate controls which region of the solution manifold GD converges to, effectively acting as a regularization hyperparameter that balances these conflicting objectives.

Results

Phase Transition: The transition from flow-aligned behavior to EoS behavior is sharp and occurs consistently at $\eta_c \approx 2/s_{GF}$ across different architectures, datasets, and loss functions.
Robustness: The observed trade-off between increasing norm and decreasing sharpness holds for various norms ( $\ell_1, \ell_2$ , nuclear) and sharpness measures (worst-case, average-case, adaptive).
Optimization Dynamics: In the EoS regime, the training trajectory oscillates around the sharpness bound $2/\eta$ , but the parameter norm grows monotonically over time, indicating the trajectory visits regions of higher complexity.
Classification vs. Regression: While the U-shaped generalization curve is prominent in many regression and classification settings, the authors note that in some specific classification scenarios (e.g., with Cross-Entropy loss), the relationship can be more complex, though the trade-off between norm and sharpness remains.

Significance and Claims

The paper argues that the prevailing view of implicit regularization as a single, static bias is insufficient to explain the generalization performance of deep learning.

Beyond Single Bias: The success of GD-based training cannot be attributed solely to norm minimization (as suggested by small learning rate theory) or solely to sharpness minimization (as suggested by EoS theory).
Dynamic Interplay: Good generalization arises from a dynamic trade-off between these competing biases, mediated by the learning rate.
Practical Implication: The learning rate should be viewed not just as a convergence speed parameter, but as a critical regularization hyperparameter that determines the structural properties (complexity vs. flatness) of the final model.
Limitations: The authors acknowledge that their theoretical results are restricted to simplified models (diagonal linear networks) due to the difficulty of characterizing implicit biases in general non-linear networks. However, the consistency of empirical results across diverse architectures suggests the phenomenon is general.

In summary, the work motivates a broader perspective on implicit regularization that accounts for the interaction between multiple biases shaped by optimization dynamics, rather than isolating them.