Regularized Random Fourier Features and Finite Element Reconstruction for Operator Learning in Sobolev Space

Technical Summary: Regularized Random Fourier Features and Finite Element Reconstruction for Operator Learning in Sobolev Space

Problem Statement
Operator learning aims to approximate mappings between infinite-dimensional function spaces, such as the solution operators of partial differential equations (PDEs). While kernel-based methods offer theoretically justified approximations with strong convergence guarantees, they suffer from computational prohibitions when applied to large training sets (due to the $O(m^2)$ complexity of the kernel matrix) and exhibit sensitivity to noise. Standard neural operators (e.g., DeepONet, FNO) address scalability but often require extensive training resources and lack explicit theoretical guarantees regarding conditioning and generalization in noisy regimes. Furthermore, many existing random feature methods assume noise-free data or require frequency-domain samples that may not be accessible. This work addresses the challenge of learning operators from noisy data while maintaining computational efficiency and theoretical stability.

Methodology
The authors propose a Regularized Random Fourier Feature (RRFF) approach coupled with a Finite Element Reconstruction map (RRFF-FEM). The framework decomposes the operator learning problem into two stages: learning a finite-dimensional map $\hat{f}$ and reconstructing the output function via a recovery map.

1. Regularized Random Fourier Features (RRFF):

   • Instead of standard minimum-norm interpolation, the method employs a frequency-weighted Tikhonov regularization. The coefficients $c^{(j)}$ for the random feature expansion are obtained by solving:
     $$c^{(j)} = \arg\min_{x \in \mathbb{R}^N} \|Ax - v^{(j)}\|_2^2 + \alpha \sum_{k=1}^N \|\omega_k\|_2^p |x_k|^2$$
     where $A$ is the random feature matrix, $v^{(j)}$ represents noisy target data, and the regularization term penalizes coefficients associated with high-frequency weights $\omega_k$. This specifically targets the suppression of high-frequency noise.
   • Feature Weights: The random feature weights $\{\omega_k\}$ are sampled from a multivariate Student's $t$ distribution with degrees of freedom $\nu$. This generalizes prior work, covering the range from Cauchy ($\nu=1$) to Gaussian ($\nu \to \infty$) distributions, allowing for flexibility in encoding function regularity.

2. Finite Element Recovery Map (FEM):

   • To handle arbitrary spatial domains and meshes, the method utilizes Lagrange finite element interpolation as the recovery map ($R_V$). This maps the vector of function evaluations (output of the random feature model) back to a continuous function in the target space $V$. This allows the learned operator to be evaluated at any point in the spatial domain, not just the training collocation points.

3. Theoretical Framework:

   • The paper establishes high-probability bounds on the extreme singular values of the random feature matrix $A$ associated with the Student's $t$ distribution.
   • It proves that if the number of features $N$ scales as $N \sim m \log m$ (where $m$ is the number of training samples), the matrix is well-conditioned with high probability. This ensures stable estimation and provides generalization guarantees.

Key Contributions

• Novel Algorithm: Introduction of RRFF-FEM, a method combining regularized random features with finite element reconstruction specifically designed for noisy operator learning.
• Theoretical Generalization: Derivation of concentration bounds for the condition number of random feature matrices where weights are drawn from multivariate Student's $t$ distributions. This unifies and extends previous results for Gaussian and Cauchy distributions.
• Conditioning Analysis: Proof that the system remains well-conditioned when the complexity ratio $N/m$ scales logarithmically with the sample size, providing a theoretical basis for the method's stability.
• Robustness to Noise: Demonstration that the frequency-weighted regularization effectively suppresses high-frequency noise, outperforming unregularized Random Fourier Features (RFF) in noisy settings.

Experimental Results
The authors conducted extensive numerical experiments on benchmark PDE problems: Advection (I, II, III), Burgers' equation, Darcy flow, Helmholtz equation, Navier-Stokes equations, and structural mechanics.

• Performance vs. Unregularized RFF: Across all benchmarks, the regularized RRFF methods consistently achieved lower relative test errors and faster training times compared to unregularized RFF. The regularization improved the conditioning of the linear system, accelerating convergence.
• Comparison with Kernel Methods: For Burgers' and Darcy flow problems, RRFF methods achieved test errors comparable to Radial Basis Function (RBF) and Matérn kernel methods but required significantly less training time (e.g., orders of magnitude faster for Darcy flow).
• Comparison with Neural Operators: While the paper notes that DeepONet and FNO results cited from prior literature (on noise-free data) show competitive accuracy, the RRFF approach achieves similar accuracy on noisy data without the high computational cost of training deep neural networks.
• Student's $t$ Distribution: The experiments utilized various degrees of freedom ($\nu = 2, 3, \infty$). The results indicated that $\nu = \infty$ (Gaussian) often yielded the lowest errors, though Student's $t$ with finite $\nu$ provided robust performance.
• Noise Resilience: Visualizations of pointwise errors confirmed that RRFF-FEM effectively reduced noise artifacts in the reconstructed solutions compared to unregularized counterparts.

Significance and Claims
The paper claims that the proposed RRFF-FEM method offers a computationally efficient and robust alternative to both kernel-based and neural operator methods for learning operators from noisy data. By leveraging random features with regularization and finite element reconstruction, the method:

1. Avoids the $O(m^2)$ computational bottleneck of kernel methods.
2. Provides theoretical guarantees on the conditioning of the learning problem.
3. Maintains competitive accuracy while significantly reducing training time.
4. Does not require specialized hardware (e.g., GPUs) or costly computational resources, making it accessible for a wide range of scientific computing applications.

The authors conclude that the method is particularly effective in regimes where data is corrupted by noise, a common scenario in practical scientific and engineering applications where governing equations may be unknown or data is imperfect.