Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers Equation

Technical Summary: Quantitative Sobolev Approximation Bounds for Neural Operators with Empirical Validation on Burgers' Equation

Problem Statement
Neural operators have emerged as resolution-invariant surrogates for solving partial differential equations (PDEs), learning mappings between infinite-dimensional function spaces. However, a critical gap exists in the theoretical understanding of their approximation properties specifically within Sobolev norms. While Sobolev norms ($H^s$) are the natural metrics for PDE well-posedness, stability, and generalization because they control both function values and derivatives, existing literature lacks rigorous, quantitative bounds on how the approximation error in these norms scales with the number of trainable parameters. Practitioners currently rely on heuristic model sizing without principled guidance on the relationship between model complexity and Sobolev-norm accuracy.

Methodology
The paper bridges functional analysis and scientific machine learning through a two-pronged approach: theoretical derivation and empirical validation.

1. Theoretical Framework:

   • Setting: The authors consider a continuous nonlinear operator $G: H^s(D) \to H^t(D')$ where inputs are restricted to a compact subset $K \subset H^s(D)$.
   • Compactness Argument: Leveraging the Rellich–Kondrachov theorem, the paper establishes that for $s > d/2$, the embedding $H^s(D) \hookrightarrow H^t(D)$ is compact. This allows the infinite-dimensional operator learning problem to be reduced to a finite-dimensional approximation task via projection onto orthonormal bases.
   • Approximation Construction: The proof constructs a neural operator $G_\theta$ by:
     1. Projecting the input $f$ onto a finite-dimensional subspace spanned by the first $N$ basis functions.
     2. Approximating the resulting finite-dimensional map using a ReLU neural network (invoking the universal approximation theorem in finite dimensions).
     3. Reconstructing the output in the target Sobolev space.
   • Complexity Bound: By analyzing the trade-off between projection error and neural network approximation error, the authors derive an explicit bound on the number of parameters $N$ required to achieve an error $\epsilon$.

2. Empirical Validation:

   • Task: The theory is tested on the solution operator of the one-dimensional viscous Burgers' equation, $G: u_0 \mapsto u(\cdot, 1)$, mapping initial conditions in a bounded $H^1$-ball to final-time solutions.
   • Architecture: Fourier Neural Operators (FNOs) are trained using an $H^1$-loss (incorporating both $L^2$ error and gradient error via periodic finite differences).
   • Experimental Design: A sweep of model sizes is conducted by varying Fourier modes and channel widths, resulting in parameter counts ranging from $\approx 7.4 \times 10^4$ to $\approx 1.8 \times 10^6$. Training is performed on synthetic data generated by a spectral solver.

Key Contributions

1. Universal Approximation in Sobolev Norms: The paper establishes that continuous operators between Sobolev spaces can be uniformly approximated on compact sets by finite-dimensional neural operators, with the approximation measured directly in the Sobolev norm.
2. Quantitative Complexity Bound: The authors derive an explicit relationship showing that to achieve an error $\epsilon$ in the $H^t$-norm, a neural operator requires $O(\epsilon^{-d/s})$ parameters. Inverting this yields an asymptotic error scaling of $\|G - G_\theta\|_{H^t} \lesssim C N^{-s/d}$, where $N$ is the parameter count.
3. Empirical Scaling Laws: Through extensive numerical experiments, the paper validates these theoretical concepts and provides data on the actual scaling behavior of FNOs, revealing the gap between worst-case theoretical bounds and practical optimization dynamics.

Results

• Sobolev Fidelity: The trained FNOs successfully learn the solution operator, achieving test $H^1$-errors as low as $O(10^{-7})$ to $O(10^{-9})$ in favorable optimization regimes. Visual inspection confirms that the models accurately predict both the solution values and their spatial derivatives, validating the "Sobolev-norm fidelity" hypothesis.
• Scaling Behavior: A log-log plot of the best-epoch Sobolev error versus parameter count reveals a power-law decay $\|G - G_\theta\|_{H^1} \approx C N^{-\alpha}$.
  • Theoretical Benchmark: For the 1D Burgers setting ($s=1, d=1$), the theory predicts $\alpha = s/d = 1$.
  • Empirical Result: The fitted exponent is $\alpha \approx 0.11$. This indicates that while increasing model size reduces error, the rate of improvement is significantly slower than the worst-case theoretical bound suggests.
• Optimization Instabilities: Long-horizon training (up to 1000 epochs) on the largest models reveals significant optimization instabilities. The largest model ($N \approx 1.8 \times 10^6$) exhibits sharp spikes in loss (increasing by orders of magnitude) after initially reaching low-error states. Consequently, using "final-epoch" error metrics produces misleading "U-shaped" curves, whereas "best-epoch" metrics better reflect the model's approximation capacity.

Significance and Claims
The paper claims to provide a principled link between architecture size, PDE regularity, and the accuracy of learned surrogates in Sobolev spaces.

• Theoretical: It confirms that the compactness of Sobolev embeddings (via Rellich–Kondrachov) underpins the feasibility of approximating infinite-dimensional operators with finite-dimensional networks, providing a rigorous foundation for operator learning in $H^s$ spaces.
• Practical: The results demonstrate that while neural operators can achieve high Sobolev accuracy, the path to this accuracy is constrained by optimization dynamics rather than just capacity. The paper argues that controlling the optimization trajectory (e.g., via early stopping or regularization) is as critical as increasing model size for reducing Sobolev error.
• Modesty: The authors acknowledge that the derived complexity bound is based on worst-case arguments. The significant discrepancy between the theoretical exponent ($\alpha=1$) and the empirical exponent ($\alpha \approx 0.11$) suggests that architectural biases and optimization landscapes play a dominant role in practice, and that refining these bounds with specific regularity assumptions (e.g., Lipschitz continuity) is a necessary future step.

In summary, the work validates that Sobolev-space approximation theory meaningfully predicts neural operator behavior while highlighting that practical scaling is heavily influenced by optimization stability, urging a shift from purely heuristic model sizing to strategies that account for both theoretical capacity and training dynamics.