Technical Summary: Solving Convolution-Type Integral Equations Using Preconditioned Neural Operators

1. Problem Statement

The paper addresses the challenge of solving large-scale, ill-conditioned linear systems arising from convolution-type integral equations. These equations appear frequently in signal processing (e.g., finite impulse response filters) and image processing (e.g., deblurring).

When discretized, these problems yield linear systems $A_n x = y$ where $A_n$ is typically a symmetric positive definite (SPD) Toeplitz or block-Toeplitz matrix. A critical difficulty with these specific systems is the mismatch between algebraic and geometric frequency components:

• Algebraic frequencies are defined by the eigenvalues of the matrix (large eigenvalues = algebraic high-frequency; small eigenvalues = algebraic low-frequency).
• Geometric frequencies refer to the oscillation of the vector entries (rapid oscillation = geometric high-frequency; smooth/slow oscillation = geometric low-frequency).

For standard Partial Differential Equation (PDE) problems, algebraic high-frequency components correspond to geometric high-frequency components. However, for convolution-type integral equations with smooth kernels, algebraic low-frequency components correspond to geometric high-frequency components.

Conventional iterative solvers, such as the Multigrid method, rely on "smoothers" (e.g., Jacobi iterations) to remove geometric high-frequency errors and "coarse-grid corrections" to remove geometric low-frequency errors. In integral equations, both the smoother and the coarse-grid correction tend to target the same algebraic high-frequency (geometric low-frequency) components, leaving the algebraic low-frequency (geometric high-frequency) errors unresolved. This causes standard multigrid methods to fail or converge very slowly for these specific problems.

2. Methodology

The authors propose a hybrid iterative algorithm that combines classical smoothers with Neural Operators to address the specific spectral properties of integral equations.

2.1. Preconditioned Training Strategy

The core innovation is a novel training strategy for neural operators designed to specifically solve the algebraic low-frequency components (which are the geometric high-frequency components in this context).

• Loss Function: Instead of minimizing the standard $L_2$ error, the authors introduce a preconditioned loss function:
  $$L_{exp}(\theta) = \mathbb{E}_{y \sim \mu} [\|P_n(N_\theta(y) - x)\|_2^2]$$
  where $N_\theta$ is the neural operator, $x = A_n^{-1}y$ is the true solution, and $P_n$ is a preconditioning matrix.
• Spectral Inversion: To force the neural operator to learn the algebraic low-frequency components, $P_n$ is chosen to have opposite spectral properties to $A_n$. Specifically, if $v$ is an eigenvector of $A_n$ with a small eigenvalue (algebraic low-frequency), $P_n$ should assign it a large weight.
• Practical Choices for $P_n$:
  1. $P_n = C(A_n)^{-1}$, where $C(A_n)$ is a circulant preconditioner (approximating $A_n$ efficiently via FFT).
  2. $P_n = aI_n - A_n$ for some $a > \max(\text{eigenvalues of } A_n)$.
• Mechanism: By weighting the error components associated with the small eigenvalues of $A_n$ heavily in the loss function, the neural network is trained to focus on correcting these specific "hard-to-solve" algebraic low-frequency modes.

2.2. Hybrid Iterative Algorithm

The trained neural operator is integrated into a two-grid-like framework (Algorithm 3.1):

1. Pre-smoothing: Apply $\kappa_1$ steps of a classical smoother (e.g., weighted Jacobi) to reduce algebraic high-frequency errors.
2. Neural Correction: Compute the residual, normalize it, and apply the trained neural operator $N_\theta$ to estimate the correction for the algebraic low-frequency components.
3. Post-smoothing: Apply $\kappa_2$ steps of the smoother to clean up any remaining errors.

This structure mirrors the multigrid method but replaces the coarse-grid correction with a neural operator specifically tuned to handle the algebraic low-frequency modes that standard smoothers miss.

3. Key Contributions

1. Algorithm Design: Development of a hybrid iterative solver that uses neural operators to target algebraic low-frequency components in convolution-type integral equations, a regime where traditional multigrid methods struggle.
2. Training Strategy: Proposal of a preconditioned loss function that inverts the spectral bias of standard neural operators, enabling them to learn high-frequency (geometric) but low-frequency (algebraic) solution components.
3. Theoretical Analysis:
   • Generalization Error: Derived an upper bound for the generalization error of a two-layer ReLU network trained with this strategy, showing convergence to zero as the number of training samples increases and optimization error decreases.
   • Convergence: Proved the convergence of the hybrid iterative algorithm under reasonable assumptions regarding the neural operator's ability to satisfy a "correcting condition" and "stability condition."
   • Training Dynamics: Analyzed how the choice of the preconditioner parameter $a$ in $P_n = aI - A_n$ affects the convergence rates of different frequency components during training.
4. Numerical Validation: Extensive experiments on 1D and 2D ill-conditioned systems demonstrating the method's superiority over existing solvers.

4. Experimental Results

The authors tested their method on large-scale linear systems ($n$ up to $1024$ in 1D and $1024^2$ in 2D) with varying condition numbers (controlled by the regularization parameter $\alpha$).

• Performance vs. Multigrid: The proposed method significantly outperformed the multigrid method.
  • Iteration Count: The proposed method required roughly 6–16 iterations regardless of the condition number, whereas the multigrid method required thousands of iterations (e.g., ~41,000 for 2D with $\alpha=10^{-5}$).
  • Computational Time: The proposed method was orders of magnitude faster (e.g., seconds vs. hours for 2D problems).
• Performance vs. PCG: The method also outperformed the Preconditioned Conjugate Gradient (PCG) method with block-circulant preconditioners in both iteration count and time.
• Robustness: The convergence rate of the proposed method remained stable as the problem size ($n$) and condition number ($\kappa$) increased, unlike the multigrid method which degraded significantly.
• Transferability: Experiments showed that a neural operator trained on a smaller system size ($n=256$) could be transferred to larger systems ($n=512, 1024$) with minimal fine-tuning, retaining high efficiency.

5. Significance and Claims

The paper claims that this approach offers a robust and efficient alternative for solving ill-conditioned convolution-type integral equations, a class of problems where standard geometric multigrid methods fail due to the spectral mismatch between algebraic and geometric frequencies.

• Efficiency: The method achieves significant speedups by leveraging the parallel nature of neural networks and the specific training strategy to target the most difficult error components.
• Generality: While focused on integral equations of the second kind, the authors claim the framework is generalizable to other types of integral equations (e.g., first kind) and does not strictly require a "good" preconditioner in the traditional numerical linear algebra sense, only a matrix with reversed spectral properties.
• Limitations: The authors acknowledge that the method requires a training phase, which is computationally expensive compared to the evaluation phase. If the underlying system changes frequently, retraining is necessary. They suggest future work on meta-learning approaches (e.g., Meta-MgNet) to solve classes of systems rather than specific instances.

In summary, the paper demonstrates that by redefining the training objective of neural operators to align with the algebraic structure of the problem, they can effectively complement classical iterative solvers to solve difficult integral equations that have historically been resistant to standard multigrid techniques.