Neural operators for solving nonlinear inverse problems

1. Problem Statement

The paper addresses the challenge of solving nonlinear inverse problems formulated as operator equations:
$$F[x] = y$$
where $F: D(F) \subseteq X \to Y$ is a nonlinear operator between infinite-dimensional function spaces (typically Sobolev or Lebesgue spaces). These problems are ill-posed, meaning solutions do not depend continuously on the data $y$, and noise ($y^\delta$) can lead to unstable solutions.

The Core Challenge:
Traditionally, solving these problems involves Tikhonov regularization using classical discretizations (e.g., Finite Element Methods - FEM) to approximate the forward operator $F$. However, in many modern applications, the operator $F$ is not explicitly known via physical laws but is specified indirectly via training data (input-output pairs). The authors aim to replace the classical discretized operator $F_n$ with a Neural Operator (specifically DeepONet architecture) learned from data, and rigorously analyze the convergence and stability of this approach within the framework of Tikhonov regularization.

2. Methodology

The paper develops a theoretical framework and numerical strategy combining Tikhonov regularization with Neural Operators.

A. Regularization Framework

The authors utilize Tikhonov regularization to stabilize the inverse problem. They seek a minimizer $x_{\alpha}^\delta$ of the functional:
$$T_{\alpha}^\delta[x] := \|F_n[x] - y^\delta\|_Y^2 + \alpha \|x - x^{(0)}\|_X^2$$
where:

• $F_n$ is the surrogate operator (Neural Operator).
• $y^\delta$ is noisy data with noise level $\delta$.
• $\alpha$ is the regularization parameter.
• $x^{(0)}$ is a prior guess (often 0).

B. Neural Operator Construction

The paper defines the Neural Operator $F_n$ based on the DeepONet architecture, which approximates operators mapping between function spaces.

• Architecture: The operator is constructed using a "branch" network (approximating functionals of the input) and a "trunk" network (approximating the output function).
• Training Strategy: Instead of treating the network as a black box, the authors propose a two-step strategy to determine network coefficients (priors):
  1. Linearization: Linearize the nonlinear operator $F$ around a reference point.
  2. Orthonormalization: Use Gram-Schmidt orthonormalization on the training data to construct a linear surrogate $F_N^\#$ explicitly.
  3. Neural Approximation: Approximate the linear functionals and output functions of $F_N^\#$ using neural networks. This allows for the predetermination of certain network parameters (e.g., sampling points, weights derived from quadrature rules), significantly simplifying the training process.

C. Extension to Sobolev and Lebesgue Spaces

A critical methodological contribution is extending the Universal Approximation Theorem for neural operators.

• Standard theorems apply to spaces of continuous functions ($C(\Omega)$).
• Inverse problems often require solutions in Sobolev spaces ($H^s$) or Lebesgue spaces ($L^2$), where pointwise evaluation is not always well-defined.
• Solution for $L^2$: The authors introduce a mollification technique. They approximate the solution $x$ with a smooth function $x_\xi$ (via convolution with a mollifier) before feeding it into the neural operator. This ensures pointwise evaluations are valid while maintaining convergence properties in the $L^2$ norm.

3. Key Contributions

1. Theoretical Convergence Rates:
   The paper derives rigorous error estimates for the regularized solution $x_{\alpha}^\delta$ when using a Neural Operator surrogate. They establish the optimal balance between:

   • The noise level ($\delta$).
   • The approximation error of the neural operator ($\rho_n$).
   • The regularization parameter ($\alpha$).
   • The mollification parameter ($\xi$) for $L^2$ spaces.
     The main result shows that with appropriate parameter choices (e.g., $\alpha \sim \delta$, $\rho_n \sim \delta$), the convergence rate is $O(\sqrt{\delta})$, matching classical results.

2. Generalization to Non-Continuous Spaces:
   The authors prove that neural operators can approximate operators defined on Sobolev spaces ($H^s$ with $s > m/2$) and Lebesgue spaces ($L^2$) by leveraging compact embedding theorems and mollification. This bridges the gap between neural operator theory (usually defined on $C(\Omega)$) and the requirements of inverse problem regularization.

3. Structured Training (Priors):
   The paper proposes a method to pre-calculate parts of the neural operator structure based on the training data geometry (orthonormalization) and classical integration rules. This reduces the complexity of the non-linear training problem and provides a theoretical basis for the network architecture.

4. Quantitative Analysis of Surrogates:
   The authors analyze the trade-off between the number of training samples ($N$), the number of network coefficients ($n$), and the approximation error, providing explicit bounds for linear and nonlinear cases.

4. Numerical Results

The authors validate their theory on two benchmark inverse problems:

• Example 3.1 (The "a-example"): A nonlinear differential equation where the coefficient $x$ is in $H^1$.
• Example 3.2 (The "c-example"): A Schrödinger-type equation where the coefficient $x$ is in $L^2$.

Key Findings from Experiments:

• Accuracy: The Neural Operator surrogate ($F_n$) generally outperforms classical Finite Element Methods (FEM) in terms of Mean Squared Error (MSE), particularly when the initial guess is far from the true solution.
• Robustness to Initialization: FEM-based optimization is highly sensitive to the initial guess. The Neural Operator approach is more robust, maintaining low error across different initializations.
• Noise Stability:
  • The linear surrogate ($F_N^\#$) is extremely stable against noise but less flexible for complex nonlinearities.
  • The trained Neural Operator ($F_n$) handles nonlinearity well but shows some oscillation under high noise due to the stochastic nature of the optimizer (AdamW).
  • FEM performance degrades significantly with noise, often requiring finer meshes that paradoxically worsen results due to overfitting the noise.
• Training Efficiency: The total training time for the neural operator (including data generation) was approximately 10 seconds for $N=20$ samples, demonstrating high efficiency for these 1D problems.
• Data Saturation: Increasing the number of training samples beyond a certain point (e.g., $N=30$ to $40$) yielded diminishing returns, suggesting the model had already captured the essential operator features.

5. Significance

• Bridging Theory and Practice: This work is significant because it moves neural operators from a "black box" heuristic to a mathematically rigorous tool for ill-posed inverse problems. It provides the necessary convergence rates and parameter selection rules required for practical deployment.
• Handling Infinite Dimensions: By extending approximation theory to Sobolev and $L^2$ spaces, the paper addresses a major limitation of previous neural operator literature, making it applicable to standard PDE-based inverse problems.
• Efficiency: The proposed "structured training" (using priors and orthonormalization) offers a pathway to reduce the computational cost and complexity of training neural operators, which is often a bottleneck.
• Alternative to FEM: The results suggest that for problems where the forward operator is expensive to simulate or unknown, learning a neural operator surrogate can provide more robust and accurate reconstructions than traditional discretization methods, provided the training data is available.

In summary, the paper establishes a solid theoretical foundation for using Neural Operators as surrogates in Tikhonov regularization, proving their convergence, extending their applicability to necessary function spaces, and demonstrating their superior performance in numerical experiments compared to classical methods.