🤖 machine learning

Beyond $\ell_2$ -norm and $\ell_\infty$ -norm: A Curvature-Inspired $\ell_p$ -Norm Scheme for Deep Neural Networks

This paper proposes a novel curvature-inspired $\ell_p$ -norm optimization scheme with a dynamically decaying $p$ parameter that transitions from suppressing high-curvature dominance to enabling stable updates, resulting in the LPSGD and LPSGDM optimizers that achieve $O(T^{-1/2})$ convergence and improved generalization across various deep neural network architectures and datasets.

Original authors: Jianhao Xu, Zhuang Yang

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianhao Xu, Zhuang Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Navigating a Shifting Landscape

Imagine you are trying to guide a heavy cart down a mountain to reach the very bottom of a valley (the perfect solution for an AI model). The terrain of this mountain is the "loss landscape" of a Deep Neural Network (DNN).

The problem is that this mountain changes shape as you travel:

At the top (Early Training): The terrain is wild and jagged. Some paths are incredibly steep cliffs (high curvature), while others are gentle, flat slopes (low curvature).
At the bottom (Late Training): The terrain smooths out. The cliffs disappear, and the ground becomes relatively flat and uniform.

The paper argues that the tools we usually use to guide the cart (optimizers) are stuck using only one type of wheel, which works well in one situation but fails in the other.

The Problem: Two Bad Wheels

The authors explain that current AI training methods generally rely on one of two "geometries" (ways of measuring distance and direction):

The ℓ2-Norm (The Standard Wheel):
- How it works: It treats every direction equally based on the size of the push.
- The Flaw: On the jagged mountain top, this wheel gets stuck. If there is a steep cliff in one direction, the wheel gets terrified and slows down everything to avoid falling. It ignores the gentle, flat slopes where it could actually move fast. It's like driving a car with sensitive brakes that lock up if you hit a single bump, preventing you from speeding up on the straightaways.
The ℓ∞-Norm (The "Sign" Wheel):
- How it works: It ignores the size of the push entirely and only looks at the direction (left or right, up or down). It takes steps of the exact same size in every direction.
- The Flaw: This works great on the jagged cliffs because it doesn't care about the steepness; it just marches forward. However, once you reach the flat valley floor, this wheel becomes useless. Because it takes the same big step everywhere, it starts bouncing back and forth (oscillating) and can't settle down gently into the very bottom of the valley.

The Solution: The "Shape-Shifting" Wheel

The authors propose a new method called ℓp-norm scheduling. Instead of choosing one wheel, they built a shape-shifting wheel that changes its shape depending on where you are on the mountain.

The Strategy: They use a "Cosine Schedule" (a smooth, wave-like curve) to change the wheel's shape over time.
- Early Days (The Jagged Mountain): The wheel starts with a shape that acts like the ℓ∞-norm. It ignores the extreme steepness of the cliffs and pushes through the rough terrain efficiently, preventing the "lock-up" problem.
- Late Days (The Flat Valley): As the training progresses, the wheel smoothly morphs into the standard ℓ2-norm shape. Now that the ground is flat, it can take precise, gentle steps to settle perfectly into the lowest point without bouncing around.

Think of it like a hiker's footwear:

In the beginning, they wear sturdy, spiked boots to grip the rocky, uneven cliffs and keep moving forward.
As they reach the flat meadow at the bottom, they switch to soft, flexible slippers to walk quietly and find the exact center of the field without tripping.

How They Proved It Works

The paper doesn't just guess; they did two things:

Math Proof: They used mathematics to prove that this "shape-shifting" method is guaranteed to find the bottom of the valley eventually, and they calculated exactly how fast it will get there.
Real-World Tests: They tested their new optimizers (named LPSGD and LPSGDM) on famous image datasets (like CIFAR and ImageNet) using standard AI models (like ResNet).
- The Result: Their "shape-shifting" method consistently beat the standard methods. It learned faster in the beginning and achieved higher accuracy at the end. For example, on one test, it improved the accuracy by nearly 2% compared to the best existing methods—a huge deal in the world of AI.

Summary

In short, the paper says: "Don't use a single tool for the whole job."

Training an AI is like traveling through a landscape that changes from a jagged mountain to a flat plain. The authors created a smart optimizer that starts with a "climb-anywhere" mode and smoothly transitions into a "precision-walk" mode. This allows the AI to learn faster and end up with a smarter, more accurate model than if it had used a single, unchanging method the whole time.

Technical Summary: Beyond ℓ2-norm and ℓ∞-norm: A Curvature-Inspired ℓp-Norm Scheme for Deep Neural Networks

Problem Statement

The training of Deep Neural Networks (DNNs) is often hindered by curvature anisotropy, a phenomenon where the loss landscape exhibits vastly different curvatures across parameter dimensions. The paper identifies that existing optimizers typically rely on fixed geometric assumptions that fail to adapt to the dynamic nature of this anisotropy during training:

ℓ2-norm based optimizers (e.g., SGD): These are dominated by high-curvature directions. In early training stages where curvature anisotropy is strong (characterized by large Hessian eigenvalue outliers), ℓ2-based updates restrict progress along flatter directions, leading to slower convergence.
ℓ∞-norm based optimizers (e.g., sign-based methods like Lion, signSGD): These update coordinates with the same magnitude based on gradient signs. While effective in highly anisotropic regions, they tend to cause oscillations in flatter regions (later training stages) due to the lack of magnitude scaling.

Recent analyses of Hessian eigenvalue distributions (e.g., in ResNet-18 on CIFAR-10 and ResNet-50 on ImageNet-1K) reveal that the loss landscape evolves from a highly anisotropic, non-convex state with extreme eigenvalues in the early epochs to a flatter, weakly anisotropic, and approximately convex region in later epochs. The central problem addressed is whether a single fixed norm geometry (ℓ2 or ℓ∞) is sufficient for the entire training process, or if a dynamic scheme is required to match these evolving curvature structures.

Methodology

The authors propose a Cosine ℓp-Norm Scheme that dynamically adjusts the norm parameter $p$ throughout the training process to align with the changing curvature anisotropy.

1. Theoretical Motivation

Based on the ℓp-smoothness model, the update direction for steepest descent in ℓp geometry involves a coordinate-wise scaling factor of $|g^{(i)}|^{p-2}$ .

When $p=2$ (Euclidean), the update scales linearly with the gradient magnitude.
When $p > 2$ , the scaling becomes sublinear ( $\alpha = \frac{1}{p-1} < 1$ ), compressing the directional imbalance and suppressing the dominance of high-curvature directions.
As $p \to \infty$ , the update approaches a sign-based form (ℓ∞), which ignores magnitude entirely.

The authors argue that a large $p$ (e.g., $p > 2$ ) is beneficial in early training to handle strong anisotropy, while a smaller $p$ (approaching 2) is preferable in later stages to ensure stable, refined updates in flatter regions.

2. The Proposed Algorithms

The scheme is integrated into Stochastic Gradient Descent (SGD) and SGD with Momentum (SGDM) to create two new optimizers: LPSGD and LPSGDM.

Dynamic Scheduling: The norm parameter $p_s$ at epoch $s$ is scheduled using a cosine annealing strategy:
$p_s = 2 + (p_{\max} - 2) \cdot \frac{1 + \cos(\pi \frac{s-1}{S-1})}{2}$
where $p_{\max} \geq 2$ is the initial maximum norm, and $S$ is the total number of epochs. The schedule starts at $p_{\max}$ and smoothly decays to 2.
Update Rule: For a stochastic gradient $g_t$ (or momentum buffer $m_t$ in LPSGDM), the update direction $u_t$ is computed via a nonlinear scaling map:
$u_t = \frac{g_t}{(|g_t| + \epsilon)^{\rho_s}}$
where $\rho_s = \frac{p_s - 2}{p_s - 1}$ and $\epsilon$ is a stability constant. The parameters are then updated as $\theta_{t+1} = \theta_t - \eta u_t$ (with weight decay added for LPSGDM).

3. Convergence Analysis

The paper provides theoretical guarantees for nonconvex stochastic optimization:

Under standard assumptions (ℓp-smoothness, unbiased gradients, bounded noise), the authors prove that both LPSGD and LPSGDM achieve a convergence rate of $O(T^{-1/2})$ for the expected squared gradient norm, where $T$ is the number of iterations.
The analysis relies on establishing conditional alignment and bounded second moments for the transformed update directions.

Key Contributions

Characterization of Curvature Evolution: The work empirically characterizes the evolution of Hessian eigenvalue distributions during DNN training, highlighting the shift from strong anisotropy to weak anisotropy.
Cosine ℓp-Norm Scheme: A novel scheduling strategy that transitions the optimizer's geometry from a high- $p$ norm (suppressing outlier curvature dominance) to the Euclidean norm (ensuring stability in flat regions) with negligible computational overhead.
New Optimizers (LPSGD/LPSGDM): The integration of this scheme into standard first-order methods, resulting in algorithms that outperform existing baselines.
Theoretical Guarantees: Rigorous proof of the $O(T^{-1/2})$ convergence rate for the proposed algorithms in nonconvex settings.

Experimental Results

Extensive experiments were conducted on benchmark datasets (CIFAR-10, CIFAR-100, ImageNet-1K) using various architectures (VGG-11, ResNet-18, ResNet-50). The proposed methods were compared against SGD w/Momentum, AdamW, and Lion.

CIFAR-10: LPSGDM consistently achieved the highest Top-1 accuracy across all models and checkpoints. For instance, on ResNet-18, it reached 94.81% accuracy at 200 epochs, outperforming SGD (93.15%), AdamW (94.09%), and Lion (94.04%).
CIFAR-100: The performance gap widened in this more challenging setting. LPSGDM achieved 77.98% on ResNet-18 and 76.78% on ResNet-50, significantly outperforming baselines (e.g., +1.87% over the best baseline on ResNet-50).
ImageNet-1K: On ResNet-50, LPSGDM surpassed all baselines in the middle and late training stages, achieving 74.04% Top-1 accuracy at epoch 60, which was a 2.25 percentage point improvement over AdamW.
Generalization: Notably, LPSGDM often achieved higher testing accuracy even when the training Negative Log-Likelihood (NLL) was not the lowest, suggesting the method improves the generalization quality of the solution rather than just minimizing training loss.

Significance and Claims

The paper claims that the proposed cosine ℓp-norm scheme effectively addresses the limitation of fixed-norm optimizers by adapting to the intrinsic geometric changes of DNN loss landscapes. By dynamically suppressing the dominance of high-curvature directions early in training and transitioning to stable Euclidean updates later, the method achieves superior generalization performance. The authors emphasize that this approach requires only first-order information and adds minimal computational cost (element-wise rescaling), making it a practical and scalable enhancement to existing optimization frameworks. The theoretical convergence rate of $O(T^{-1/2})$ confirms that these empirical gains are supported by rigorous mathematical analysis.