Performance of Neural and Polynomial Operator Surrogates

1. Problem Statement

The paper addresses the challenge of constructing efficient surrogate operators for parameter-to-solution maps arising from parametric Partial Differential Equations (PDEs). In many scientific applications (e.g., stochastic simulation, inverse problems, optimization), the forward model $G: X \to Y$ is computationally expensive to evaluate repeatedly. The goal is to learn a surrogate $\tilde{G}$ that approximates $G$ with high accuracy while being computationally cheap to evaluate, given a limited number of training samples.

The authors focus on the trade-offs between Neural Operators (data-driven, flexible architectures) and Polynomial Surrogates (theory-driven, structured approximations), specifically investigating how their performance depends on the regularity (smoothness) of the input parameter fields.

2. Methodology

The study conducts a systematic empirical comparison of five distinct surrogate methodologies applied to two test problems: a linear parametric diffusion equation and a nonlinear hyperelasticity problem.

A. Neural Operator Surrogates

1. Reduced-Basis Neural Operators (RBNO): These encode inputs and outputs into low-dimensional coefficient spaces (via PCA/POD) and use a feedforward neural network to learn the map between coefficients.
   • Training: Two formulations are compared:
     • $L^2_\mu$ training: Minimizes standard mean squared error.
     • $H^1_\mu$ training: Incorporates derivative information (Fréchet derivatives) into the loss function to improve accuracy in downstream tasks requiring gradients.
2. Fourier Neural Operator (FNO): A discretization-invariant architecture that learns maps in the Fourier domain using spectral convolutions. It is trained using the standard $L^2_\mu$ objective.

B. Polynomial Operator Surrogates

These methods approximate the coefficient map using global polynomial expansions, leveraging reduced-basis encoders/decoders.

1. Reduced-Basis Sparse-Grid (RB-SG): Uses sparse-grid interpolation to mitigate the curse of dimensionality. It relies on a specific set of interpolation nodes and does not require non-convex optimization.
2. Reduced-Basis Tensor-Train (RB-TT): Represents the high-dimensional tensor of collocation values in a low-rank Tensor-Train (TT) format. It utilizes the ALS-cross algorithm to construct the surrogate directly from PDE solutions without forming the full tensor.

C. Experimental Design

• Input Regularity: Inputs are generated with algebraically decaying spectral coefficients, controlled by a smoothness parameter $s$.
  • Rough inputs: $s \leq 1$ (slow decay, high-frequency content).
  • Smooth inputs: $s \geq 2$ (fast decay, low-frequency content).
• Evaluation Metrics:
  • Accuracy: $L^2_\mu$ error (solution) and $H^1_\mu$ error (derivatives).
  • Cost: Decomposed into Data Generation (number of PDE solves), Setup/Training Time, and Online Evaluation Time.
• Comparison Strategy: Instead of comparing single "best" models, the authors analyze ensembles of models with varying hyperparameters to generate Pareto frontiers of cost versus accuracy.

3. Key Contributions

1. Systematic Benchmarking: Unlike previous studies focusing only on neural architectures, this work provides a fair comparison between modern neural operators and classical polynomial methods (Sparse Grids and Tensor Trains) under a unified framework.
2. Regularity-Dependent Analysis: The study explicitly demonstrates that the "best" method is not universal but depends critically on the smoothness of the input data ($s$).
3. Derivative-Informed Training: The paper quantifies the benefits of $H^1_\mu$ training for RBNOs, showing it significantly improves data efficiency, particularly when Jacobian information is available.
4. Comprehensive Cost Breakdown: The analysis separates offline costs (data generation, training) from online costs (evaluation), providing a holistic view of computational efficiency for different application scenarios.

4. Key Results

A. Impact of Input Smoothness ($s$)

• Smooth Inputs ($s \geq 2$):
  • Polynomial surrogates (RB-SG and RB-TT) are superior. They achieve significantly better data efficiency (lower error for the same number of training samples) and faster convergence rates, aligning with theoretical predictions.
  • RB-SG offers the best balance of low offline cost, simplicity, and high accuracy.
  • RB-TT is slightly more data-efficient than RB-SG but suffers from numerical instability at very high ranks and higher setup costs.
• Rough Inputs ($s \leq 1$):
  • Fourier Neural Operator (FNO) outperforms all other methods, exhibiting the fastest convergence rates.
  • Polynomial methods struggle with rough inputs due to the curse of dimensionality and the difficulty of approximating non-smooth functions with global polynomials.
  • Standard RBNOs ($L^2_\mu$) struggle, but Derivative-Informed RBNOs ($H^1_\mu$) become competitive in the low-data regime.

B. Training Formulations

• $H^1_\mu$ vs. $L^2_\mu$: Training with derivative information consistently improves data efficiency for RBNOs. While the training process is more expensive (requiring Jacobian computation), the resulting model requires fewer PDE solves to reach a target accuracy, making it advantageous when data generation is the bottleneck.

C. Computational Costs

• Offline (Setup):
  • RB-SG is the fastest to set up (no iterative training).
  • Neural Operators have high setup costs due to iterative optimization.
  • RB-TT setup is moderate but scales poorly with high accuracy requirements due to the ALS-cross algorithm complexity.
• Online (Evaluation):
  • RB-SG and RB-TT are highly efficient for large batch sizes.
  • FNO has a high fixed cost but saturates quickly; it is less efficient for small batch sizes but competitive for large ones.
  • RBNOs scale well with batch size.

5. Significance and Conclusion

The paper concludes that there is no universally superior surrogate method. The choice of methodology must be matched to the specific characteristics of the problem:

1. For Smooth Problems: Polynomial surrogates (specifically Sparse Grids) are the preferred choice due to their superior data efficiency, theoretical convergence guarantees, and lower computational overhead.
2. For Rough/High-Frequency Problems: Neural operators, particularly FNO, are necessary to capture complex features that polynomial methods miss.
3. For Gradient-Dependent Tasks: If derivative information is available, $H^1_\mu$ training for RBNOs provides a robust alternative, especially in data-scarce regimes.

This work provides a crucial guide for practitioners in the field of scientific machine learning, emphasizing that "operator learning" is not a one-size-fits-all solution and that classical approximation theory remains highly relevant and often superior for smooth parametric PDEs.