Generalization Bounds for Physics-Informed Neural Networks for the Incompressible Navier-Stokes Equations

1. Problem Statement

The paper addresses the theoretical gap in understanding the generalization error of Physics-Informed Neural Networks (PINNs) when applied to non-linear Partial Differential Equations (PDEs), specifically the incompressible Navier-Stokes equations.

While PINNs have shown empirical success in solving complex fluid dynamics problems, rigorous theoretical guarantees regarding their ability to generalize from training data (collocation points) to the true solution remain scarce, particularly for non-linear operators. Existing literature often focuses on linear PDEs or provides bounds that explicitly depend on the network width (number of parameters), leading to the "curse of dimensionality" or impractical sample complexity estimates.

The core challenge is to derive a finite-sample generalization bound for the PINN risk that:

1. Applies to the non-linear Navier-Stokes equations.
2. Is dimension-independent (does not scale exponentially with the spatial dimension $d$).
3. Does not explicitly depend on the network width (number of neurons).
4. Characterizes the generalization gap in terms of physical parameters (e.g., kinematic viscosity).

2. Methodology

The authors employ Rademacher complexity, a measure from statistical learning theory used to bound the difference between empirical risk and population risk.

A. Mathematical Setup

• Target PDE: The $(d+1)$-dimensional incompressible Navier-Stokes equations, involving velocity $u$, pressure $p$, and kinematic viscosity $\nu$.
• Network Architecture: A depth-2 neural network (one hidden layer) mapping space-time coordinates $(x, t)$ to velocity and pressure fields.
• Loss Function: A composite PINN loss consisting of:
  1. Residual Loss ($\ell_{res}$): Enforces the Navier-Stokes PDE at interior collocation points.
  2. Boundary/Initial Condition Loss ($\ell_0$): Enforces initial conditions at boundary points.
  3. Regularization: Includes terms for incompressibility ($\nabla \cdot u = 0$) and initial condition matching.
• Weight Constraints: The analysis assumes the network weights belong to a specific class $\mathcal{C}$ where certain combinations of weights (related to the derivatives of the activation function) are bounded by constants $B_{f_i}$. This allows the bounds to be independent of the network width.

B. Theoretical Derivation

The proof strategy involves decomposing the generalization error into Rademacher complexities of the residual and boundary terms:

1. Decomposition: Using Lemma 3, the total generalization error is bounded by the sum of the Rademacher complexities of the residual loss and the initial condition loss.
2. Expansion: The authors expand the partial derivatives of the neural network output (velocity and pressure) required for the Navier-Stokes loss (e.g., $\partial_t u$, $(u \cdot \nabla)u$, $\nabla^2 u$) in terms of the network weights and activation function derivatives (Lemma 4).
3. Contraction Principles: To handle the non-linear terms (specifically the convective term $(u \cdot \nabla)u$), the authors develop novel variations of the Talagrand contraction lemma (Lemmas 6 and 7). These lemmas allow the reduction of complex non-linear Rademacher complexities into those of linear function classes, provided the activation functions satisfy specific Lipschitz and boundedness conditions.
4. Final Bound: By applying standard Rademacher bounds for linear models, they derive an upper bound that depends on the number of training points ($N_r, N_0$), viscosity ($\nu$), and properties of the activation function, but not on the network width.

3. Key Contributions

1. First-of-its-kind Bounds for Non-linear PDEs: This work provides the first rigorous upper bounds on the generalization error of PINNs specifically for the Navier-Stokes equations, a highly non-linear system. Previous bounds were largely restricted to linear or second-order PDEs.
2. Dimension-Independence: The derived sample complexity bounds are independent of the spatial dimension $d$. This is a significant improvement over previous works (e.g., De Ryck et al., 2023) where bounds scaled poorly with dimension.
3. Width-Independence: The bounds do not explicitly depend on the number of parameters (width) of the neural network, provided the weights satisfy specific norm constraints. This aligns with the phenomenon of over-parameterization in deep learning.
4. Physical Parameter Characterization: The generalization gap is explicitly characterized by the kinematic viscosity ($\nu$). The bound grows linearly with viscosity, suggesting that higher viscosity (smoother flows) leads to better generalization guarantees.
5. Activation Function Recommendations: The theoretical analysis suggests that specific activation functions (e.g., $\tanh^3(x)$, $\text{sigmoid}^k(x)$) satisfy the necessary smoothness and boundedness conditions to achieve these tight bounds.

4. Results

Theoretical Results

The main theorem (Theorem 1) states that the expected worst-case generalization error is bounded by:
$$E[\text{Gap}] \leq \frac{C_r}{\sqrt{N_r}} + \frac{C_0}{\sqrt{N_0}}$$
Where:

• $N_r$ and $N_0$ are the number of interior and boundary collocation points.
• $C_r$ and $C_0$ are constants depending on the Lipschitz constants of the loss and activation functions, weight bounds, and viscosity $\nu$.
• Crucially, $C_r$ scales linearly with $\nu$.

Empirical Validation

The authors validated their theory using the Taylor-Green vortex benchmark (a standard test case for Navier-Stokes solvers).

• Setup: They trained depth-2 PINNs with $\tanh$ and $\tanh^3$ activation functions on varying numbers of collocation points ($N_r$).
• Findings:
  • There is a strong correlation (Pearson coefficient $\approx 0.90$) between the theoretical upper bound derived in Theorem 1 and the empirically measured generalization error.
  • The $\tanh^3$ activation function yielded higher correlations than standard $\tanh$, supporting the theoretical preference for higher-order smooth activations.
  • The bounds correctly predicted the scaling behavior of the error as the number of data points increased.

5. Significance and Implications

• Theoretical Foundation: This work bridges the gap between the empirical success of PINNs in fluid dynamics and statistical learning theory, providing a mathematical justification for why PINNs generalize well.
• Guidance for Architecture Design: The results suggest that using smoother, higher-order activation functions (like $\tanh^3$) and enforcing weight constraints can improve generalization guarantees.
• Sample Efficiency: The dimension-independent nature of the bound implies that PINNs are theoretically capable of solving high-dimensional fluid dynamics problems without suffering from the curse of dimensionality, provided sufficient data is available.
• Future Directions: The authors identify that extending these bounds to deeper networks (beyond depth-2) and establishing lower bounds on network size for approximation are critical next steps. They also note the potential for applying these techniques to transformer-based architectures (like AB-UPT) used in modern scientific AI.

In summary, this paper establishes a rigorous theoretical framework for understanding the generalization capabilities of PINNs in solving the Navier-Stokes equations, offering new insights into the interplay between physical parameters (viscosity), network architecture, and data requirements.