💰 quantitative finance

Convergence of the generalization error for deep gradient flow methods for PDEs

This paper establishes a rigorous mathematical foundation for deep gradient flow methods in solving high-dimensional PDEs by proving that the generalization error converges to zero as both the number of neurons and the training time approach infinity, through the decomposition of error into vanishing approximation and training components.

Original authors: Chenguang Liu, Antonis Papapantoleon, Jasper Rou

Published 2026-02-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenguang Liu, Antonis Papapantoleon, Jasper Rou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to solve a massive, incredibly complex puzzle. This puzzle represents a Partial Differential Equation (PDE). In the real world, PDEs are the "laws of physics" that describe how heat spreads, how stock prices fluctuate, or how fluids flow.

The problem? These puzzles often have too many dimensions (variables) for traditional math methods to handle. It's like trying to solve a Rubik's Cube that has a billion faces instead of six.

Enter Deep Gradient Flow Methods (DGFMs). This paper is essentially a "proof of concept" that says: "Yes, we can use Artificial Intelligence (specifically Neural Networks) to solve these massive puzzles, and here is the mathematical guarantee that it will actually work."

Here is the breakdown of the paper using simple analogies:

1. The Goal: Solving the Puzzle

The authors want to prove that if you give a neural network enough time and enough "brain cells" (neurons), it will eventually find the exact correct answer to the physics puzzle, not just a lucky guess.

They break the "mistake" (error) the computer makes into four buckets:

The Measurement Error: Did we measure the puzzle pieces correctly? (Quadrature error)
The Time-Step Error: Did we move too fast between steps? (Time-stepping error)
The Approximation Error: Is the computer's "brain" big enough to even hold the answer?
The Training Error: Did the computer actually learn the answer, or did it just give up halfway through?

The paper focuses on the last two: Can the brain hold the answer? and Will it learn it?

2. Part One: The "Universal Sketchpad" (Approximation Error)

The Question: Can a neural network actually draw the solution to these complex physics equations?

The Analogy: Imagine you have a blank canvas and a set of paintbrushes. You want to paint a perfect picture of a stormy ocean.

Old View: Maybe a few brushes aren't enough. Maybe you need a specific type of paint.
The Paper's Discovery: The authors prove that if you have enough brushes (neurons) and you are allowed to mix them in any way you want, you can paint any shape, no matter how complex, with perfect accuracy.

They call this the Universal Approximation Theorem. They showed that even though these equations happen in infinite space (like the whole ocean), a neural network with enough neurons can approximate the solution perfectly. As you add more neurons, the "paint" gets closer and closer to the real picture until the difference is zero.

3. Part Two: The "Infinite Marathon" (Training Error)

The Question: Once the computer has the right tools (neurons), will it actually learn the solution if we let it train for a long time?

The Analogy: Imagine a hiker trying to find the lowest point in a massive, foggy valley (the solution).

The Method: The hiker uses a "Gradient Flow." This means they just keep walking downhill. If they feel the ground sloping down, they step that way.
The Problem: In deep learning, the hiker is actually a swarm of thousands of tiny ants (neurons) working together. As the swarm gets bigger and bigger (approaching "infinite width"), the chaotic movement of individual ants smooths out into a single, predictable flow.

The Paper's Discovery:
The authors modeled this "swarm" mathematically. They proved that:

The Flow is Real: As the number of neurons goes to infinity, the chaotic training process turns into a smooth, predictable river flowing downhill.
No Dead Ends: They proved that this river doesn't get stuck in a small puddle (a local minimum). It flows all the way down to the absolute bottom of the valley (the global minimum), which is the true solution to the physics equation.
Time Wins: If you let the hiker walk long enough (training time $\to$ infinity), they will always reach the bottom.

4. The Grand Conclusion

The paper combines these two findings:

Capability: A big enough neural network can represent the solution.
Learning: If you train it long enough, the learning process will find that solution.

The Result: The "Generalization Error" (the difference between the computer's answer and the real truth) shrinks to zero.

Why This Matters

Before this paper, people used Deep Learning for physics problems because it "seemed to work" in experiments. It was like using a magic wand and hoping it worked.

This paper provides the mathematical spellbook. It says: "If you follow these rules (reasonable assumptions about the physics), and you use enough neurons and enough time, the magic wand is guaranteed to work."

It gives engineers and scientists the confidence to use these AI methods for critical tasks like designing new airplanes, predicting climate change, or pricing complex financial derivatives, knowing that the math backs them up.

1. Problem Statement

The paper addresses the mathematical convergence of Deep Gradient Flow Methods (DGFMs), also known as Deep Ritz methods, for solving high-dimensional Partial Differential Equations (PDEs). While DGFMs have shown empirical success in tackling high-dimensional problems where classical methods (finite difference/element) fail, a rigorous theoretical foundation for their convergence was lacking.

The authors aim to prove that the generalization error ( $E_{gen}$ ) of DGFMs converges to zero. The generalization error is defined as the difference between the true solution of the PDE ( $u^\star$ ) and the solution computed by the neural network ( $u^\star_{\theta, n}$ ).

The error is decomposed into four components:

Quadrature Error ( $E_{quad}$ ): Approximation of integrals via Monte Carlo.
Time-stepping Error ( $E_{step}$ ): Discretization error of the time scheme.
Approximation Error ( $E_{approx}$ ): The ability of the neural network architecture to represent the true solution.
Training Error ( $E_{train}$ ): The ability of the optimization algorithm (Gradient Descent/Flow) to find the global minimum of the loss function.

The paper focuses specifically on proving the convergence of the approximation error and the training error as the number of neurons ( $n$ ) and training time ( $t$ ) tend to infinity.

2. Methodology

The authors employ a multi-step analytical framework combining PDE theory, variational calculus, and infinite-width neural network theory.

A. Problem Formulation

The PDE is formulated as an evolution equation:
$u_t + Au = 0, \quad u(0) = u_0$
where the operator $A$ is split into a self-adjoint linear part $L$ and a remainder $F$ ($Au = Lu + F(u)$). The PDE is discretized in time using the Backward Euler scheme, transforming the problem into a sequence of energy minimization problems at each time step $k$ :
$U_k = \arg\min_{v} I_k(v)$
where $I_k(v)$ is an energy functional involving the $L^2$ norm of the residual, the bilinear form associated with $L$ , and the term $F$ .

B. Approximation Error Analysis (Section 3)

To prove $E_{approx} \to 0$ , the authors establish:

Well-posedness: They prove the existence and uniqueness of weak solutions to the PDE under specific assumptions (Continuity, Gårding inequality, Self-adjointness, and Lipschitz continuity of $F$ ).
Variational Equivalence: They show that the time-discretized PDE solution is equivalent to the unique minimizer of the energy functional $I_k$ .
Universal Approximation Theorem (UAT) for $\mathbb{R}^d$ : Unlike standard UATs defined on bounded domains, the authors prove a tailored version of the UAT for functions in the Sobolev space $H^1_0(\mathbb{R}^d)$ . They define a specific class of neural networks with compactly supported smooth activation functions ( $C^\infty_c$ ) and prove that this class is dense in $H^1_0(\mathbb{R}^d)$ .
Convergence: By combining the density of neural networks with the continuity of the energy functional, they demonstrate that as the number of neurons $n \to \infty$ , the neural network minimizer converges to the true PDE solution.

C. Training Error Analysis (Section 4)

To prove $E_{train} \to 0$ , the authors analyze the training dynamics in the "wide network limit" ( $n \to \infty$ ):

Gradient Flow Dynamics: They model the training process of the neural network parameters $\theta_n$ as a gradient flow. As $n \to \infty$ , the discrete dynamics of the parameters converge to a continuous infinite-dimensional Ordinary Differential Equation (ODE) (a gradient flow in function space).
Kernel Definition: They define a specific kernel $Z(x, y)$ derived from the expected outer product of the gradients of the network with respect to its parameters. This kernel differs from the standard Neural Tangent Kernel (NTK) because it incorporates the specific structure of the PDE loss functional.
Long-time Behavior: They prove that the solution to this limiting gradient flow converges to the global minimizer of the loss functional as training time $t \to \infty$ . This relies on showing that the operator governing the flow is self-adjoint, positive definite, and trace-class, allowing for spectral decomposition and exponential decay of the error.

3. Key Contributions

Rigorous Convergence Proof: The paper provides the first comprehensive convergence proof for DGFMs that explicitly links the number of neurons and training time to the vanishing of the generalization error.
Tailored Universal Approximation Theorem: The authors extend the UAT to the Sobolev space $H^1_0(\mathbb{R}^d)$ with compactly supported activation functions, addressing the specific requirements of unbounded domains common in PDE applications (e.g., option pricing).
Wide Network Limit Analysis for PDEs: They derive the specific gradient flow equation governing the training of DGFMs in the infinite-width limit and prove its convergence to the global optimum, accounting for the non-standard loss landscape of PDE residuals.
Error Decomposition: The work clarifies the distinct roles of approximation and training errors, showing that both can be driven to zero under verifiable assumptions on the PDE operators.

4. Key Results

Theorem 3.11 (Density): The space of neural networks with specific smooth, compactly supported activations is dense in $H^1_0(\mathbb{R}^d)$ .
Theorem 3.16 (Approximation Convergence): If the number of neurons $n \to \infty$ , the neural network approximation of the energy minimizer converges to the true minimizer in the $H^1_0$ norm.
Theorem 4.1 (Wide Limit Convergence): As $n \to \infty$ , the training dynamics of the finite-width network converge to the infinite-dimensional gradient flow defined by the kernel $Z$ .
Theorem 4.3 (Training Convergence): As training time $t \to \infty$ , the solution of the infinite-width gradient flow converges to the global minimizer of the loss functional.
Main Conclusion: Under reasonable assumptions (Gårding inequality, self-adjointness, Lipschitz continuity), the generalization error $E_{gen} \to 0$ as $n \to \infty$ and $t \to \infty$ .

5. Significance

Theoretical Validation: This work moves DGFMs from a heuristic "black box" approach to a mathematically grounded method, providing confidence in their application to critical scientific and engineering problems.
High-Dimensional Applicability: By proving convergence for high-dimensional PDEs (where classical grid-based methods suffer from the curse of dimensionality), the paper validates the use of deep learning for complex models in finance (e.g., Heston model, Merton model), physics, and biology.
Guidance for Practice: The results suggest that to achieve high accuracy, practitioners should ensure sufficient network width (to minimize approximation error) and sufficient training time (to minimize training error), provided the underlying PDE satisfies standard stability conditions.
Differentiation from PINNs: The paper distinguishes DGFMs from Physics-Informed Neural Networks (PINNs) by focusing on energy minimization (variational formulation) rather than residual minimization, offering a different theoretical pathway for convergence analysis.

In summary, the paper establishes that Deep Gradient Flow methods are not just empirically effective but are theoretically sound, converging to the true solution of high-dimensional PDEs as the model capacity and training duration increase.