🔢 mathematics

Global Convergence and Error Propagation in Neural Gradient Flows: A Riemannian Optimization Framework

This paper establishes a Riemannian optimization framework for neural network training that reformulates minimizing movement steps as flows on smooth submanifolds, proving global linear convergence to a unique minimizer under specific geometric conditions and demonstrating that inexact Gauss-Newton-type solvers achieve superior trajectory accuracy with fewer iterations compared to first-order baselines.

Original authors: Shixin Zheng, Yiwei Wang, Haizhao Yang

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shixin Zheng, Yiwei Wang, Haizhao Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Navigating a Foggy Mountain

Imagine you are trying to find the very bottom of a vast, foggy mountain valley (the "global minimum"). You can't see the whole valley, and the ground is uneven.

In machine learning, we usually try to find this bottom by taking small steps downhill. This paper proposes a new, smarter way to take those steps, specifically when using Neural Networks (complex computer programs that learn patterns).

The authors call their method the "Minimizing Movement Scheme" (MMS). Think of MMS not as a single giant leap, but as a series of tiny, careful steps where, at each step, you solve a mini-puzzle to find the best place to land before moving on.

The Problem: The "Jagged" Terrain

Usually, when we train a neural network, we treat the parameters (the numbers inside the computer) like they are on a flat, smooth sheet of paper. But the authors argue that the "landscape" of a neural network is actually more like a curved, crumpled sheet of rubber floating in a higher-dimensional space.

If you try to walk on this crumpled sheet using standard "flat" walking rules (standard gradient descent), you might get stuck or take inefficient paths because you aren't respecting the curve of the sheet.

The Solution: Walking on the Curve

The paper introduces a geometric framework to fix this. Here is how they do it:

1. The "Increment" Trick (The Step-by-Step Map)
Instead of asking, "Where is the absolute best spot in the whole world?", the paper asks, "If I am standing here right now, what is the best tiny step I can take?"
They call this tiny step an "increment."

Analogy: Imagine you are hiking. Instead of trying to map the whole mountain, you only look at the ground immediately around your feet. You ask, "If I move 1 inch, which direction is best?"

2. The "Manifold" (The Shape of the Path)
The authors prove that all these possible "tiny steps" form a smooth, curved surface (a Riemannian manifold).

Analogy: Think of the neural network's possible moves not as a chaotic mess, but as a smooth, curved slide. Even though the math inside the computer is messy, the shape of the possible moves is actually very orderly and smooth.

3. The "Gauss-Newton" Compass
To walk down this curved slide, you need a special compass. The authors show that a specific mathematical tool called the Gauss-Newton method acts exactly like a compass that knows how to walk on this curved slide.

Analogy: Standard methods (like Adam or L-BFGS) are like walking with a compass that assumes the ground is flat. If the ground is curved, you might walk in circles. The Gauss-Newton method is like a compass that knows the ground is curved and adjusts your path to follow the curve perfectly.

The Main Results: Why This Matters

1. It Converges Faster and More Reliably
The paper proves mathematically that if you use this "curved compass" (Gauss-Newton) to solve the mini-puzzle at each step, you are guaranteed to move toward the bottom of the valley very quickly.

The Claim: The error (how far you are from the bottom) shrinks exponentially fast. It's like sliding down a smooth slide rather than stumbling down a rocky hill.

2. Handling "Imperfect" Steps
In real life, we can't solve the mini-puzzle perfectly every time; we stop after a few seconds. The paper proves that even if your steps are slightly "imperfect" (inexact), as long as you keep taking them, you will still stay on track.

The Claim: You don't need to be perfect at every single step. As long as you are "close enough" at each step, the whole journey will still lead you to the bottom of the valley, staying within a predictable distance of the true best solution.

3. The "Trajectory Budget"
The authors also calculated a "budget" for how far the parameters can wander. They proved that the path the neural network takes stays within a safe, bounded area and doesn't go off into infinity.

Analogy: It's like having a leash on a dog. Even if the dog runs around, the leash ensures it never runs too far away from the owner.

What the Experiments Showed

The authors tested this on several tasks, like predicting numbers (regression) and recognizing images (MNIST).

The Result: Their method (using the Gauss-Newton compass) reached lower error rates and stayed closer to the "ideal" path than popular methods like Adam or L-BFGS.
Key Observation: In many cases, their method needed fewer steps to get a good result, even though each individual step took a bit more computing power. It was a trade-off: fewer, smarter steps vs. many, simpler steps.

Summary

This paper provides a mathematical "rulebook" for walking down a curved mountain using a neural network. It proves that if you treat the network's moves as steps on a smooth, curved surface (a manifold) and use a specific type of "curved compass" (Gauss-Newton), you are guaranteed to find the best solution quickly and reliably, even if your steps aren't perfect. It turns a messy, chaotic optimization problem into a clean, geometric journey.

Technical Summary: Global Convergence and Error Propagation in Neural Gradient Flows

Problem Statement
The paper addresses the global convergence of neural network optimization within the Minimizing Movement Scheme (MMS) framework. While MMS provides a robust variational time-discretization of gradient flows in infinite-dimensional Hilbert spaces, practical implementation requires restricting the search to a finite-dimensional neural network model class. This restriction transforms the convex MMS subproblem into a highly non-convex optimization problem in the parameter space. The core challenge is to establish a theoretical guarantee that iterative neural MMS steps, solved approximately, converge globally to the minimizer of the underlying energy functional, despite the non-convexity of the neural parameterization and the accumulation of errors from finite-time inner solvers and neural approximation.

Methodology
The authors develop a geometric convergence theory by reformulating the neural MMS step not as an optimization over the full parameter space, but as an optimization over a local set of increments in the ambient Hilbert space $H$ .

Increment Manifold Formulation: Instead of optimizing $w \in \mathbb{R}^p$ directly, the method optimizes the increment $h = u_{NN}(w) - u_{NN}(w_n)$ in the function space. Under assumptions of $C^2$ regularity and local non-degeneracy of the Jacobian $D u_{NN}$ , the set of feasible increments is shown to form a boundaryless smooth embedded Riemannian submanifold $\mathcal{M}$ of the Hilbert space $H$ .
Riemannian Gradient Flow Equivalence: The paper demonstrates that a natural preconditioned gradient flow in the parameter space (specifically, a Gauss–Newton-type flow using the preconditioner $(D u_{NN}^* D u_{NN})^{-1}$ ) induces exactly the Riemannian gradient flow of the increment objective on the manifold $\mathcal{M}$ . This connects second-order preconditioning to intrinsic geometric dynamics.
Geodesic Convexity Analysis: By imposing a strict interior-localization condition (ensuring the solution remains within a compact subset of the local chart) and an explicit data condition relating the step size, Jacobian non-degeneracy, and energy landscape, the authors prove that the subproblem objective is geodesically strongly convex on the reached sublevel set. This allows for the application of Riemannian optimization convergence theory.
Error Propagation Analysis: The framework quantifies the propagation of two error sources through MMS iterations:
- Finite-time inexactness: Errors from stopping the inner Riemannian solver (continuous flow or discrete descent) early.
- Neural approximation error: The discrepancy between the exact MMS proximal map and the neural approximation.
  Using the contraction property of the proximal map $J_\tau$ , the authors show that these errors are absorbed at each step, leading to a uniform tracking bound.

Key Contributions

Geometric Structure of Neural Increments: The paper establishes that the image of a local parameter ball under a neural network map forms a smooth Riemannian submanifold in the function space, provided the Jacobian is locally non-degenerate.
Equivalence of Preconditioning and Geometry: It proves that Gauss–Newton preconditioning in parameter space is mathematically equivalent to performing steepest descent with respect to the ambient Hilbert metric on the increment manifold.
Global Convergence under Inexactness: The authors derive explicit conditions (strict interior localization and data-dependent curvature bounds) under which the Riemannian gradient flow converges exponentially to the unique subproblem minimizer.
Uniform Tracking Bound: The main theoretical result (Theorems 27 and 28) shows that if the inner solver achieves sufficient accuracy (controlled by time $T$ or iteration count $K$ ) and the neural approximation error is bounded, the sequence of neural iterates tracks the exact MMS trajectory uniformly and converges to an $O(\delta)$ -neighborhood of the global minimizer.
Parameter-Space Convergence: The theory is lifted back to parameter space, providing a finite trajectory length budget that ensures the parameter iterates remain within a bounded region where the non-degeneracy condition holds, guaranteeing convergence to a limit point.

Results

Theoretical: The paper provides a rigorous proof of global convergence for the neural MMS scheme. It establishes that the inexact neural iterates converge to an $O(\delta)$ -neighborhood of the true minimizer, where $\delta$ depends on the precision of the inner solver and the neural approximation capability. The convergence rate is linear (exponential in continuous time) with a rate determined by the geodesic strong convexity parameter $\nu$ .
Numerical: Experiments on nonlinear regression (1D and 10D), California Housing data, MNIST classification, and a latent diffusion testbed indicate that the Gauss–Newton-type inner solver achieves smaller trajectory tracking errors compared to first-order baselines (Adam, L-BFGS, etc.).
- In underparameterized regimes, the Gauss–Newton method maintains a well-conditioned Jacobian (minimum singular value $\sim 10^{-2}$ ), whereas first-order methods often drive the Jacobian to near-singularity ( $\sim 10^{-14}$ ).
- The Gauss–Newton solver achieves comparable or better final energy with substantially fewer inner iterations (e.g., 5 steps vs. 500 for Adam), suggesting higher efficiency in resolving the proximal subproblem.

Significance and Claims
The paper claims to provide a geometric convergence theory for neural optimization that operates at fixed, finite width, distinguishing it from Neural Tangent Kernel (NTK) or mean-field approaches that rely on infinite-width limits. By viewing the optimization through the lens of Riemannian geometry on the increment manifold, the authors bridge the gap between the convexity of the underlying energy functional in function space and the non-convexity of the neural parameterization.

The significance lies in:

Theoretical Rigor: Offering a non-asymptotic convergence guarantee for neural MMS that accounts for both optimization inexactness and model approximation error, without requiring over-parameterization.
Algorithmic Insight: Identifying that the Gauss–Newton preconditioner is the natural geometric choice for MMS subproblems, as it respects the manifold structure of the neural model class.
Practical Validation: Demonstrating that this geometric perspective translates to practical benefits, where second-order methods maintain better numerical conditioning and track the true gradient flow trajectory more faithfully than standard first-order optimizers.

The authors remain modest regarding limitations, noting that the theory assumes $L^2$ -type functionals (excluding higher-order derivatives like $H^1$ norms where Lipschitz constants may explode) and relies on a "warm start" or small initial gap to ensure the trajectory stays within the non-degenerate local chart. They also acknowledge that the analysis of discretized Gauss–Newton updates with Levenberg–Marquardt damping remains an open problem for future work.