Robust Deep FOSLS for Transmission Problems

1. Problem Statement

The paper addresses transmission problems in heterogeneous media, specifically governed by diffusion equations with discontinuous material coefficients (e.g., varying thermal conductivity $\kappa$).

• Challenges:
  • Regularity Loss: Discontinuous coefficients cause jumps in the gradient or flux, reducing the solution's regularity (often $u \in H^1 \setminus H^2$).
  • Standard PINN Limitations: Physics-Informed Neural Networks (PINNs) relying on strong forms fail because the residual contains Dirac delta singularities, violating $L^2$ assumptions.
  • Weak Form Limitations: Variational methods like VPINNs and Deep Ritz suffer from gradient variance that does not decrease as the solution improves, requiring excessive quadrature points for stability.
  • High Contrast Sensitivity: Standard loss functionals often decouple from the true error when material contrast ($\kappa_{max}/\kappa_{min}$) is high, leading to inaccurate solutions despite low loss values.
  • Quasi-Gibbs Phenomenon: Smooth activation functions (e.g., tanh) struggle to approximate discontinuities, causing spurious oscillations (overshoots/undershoots) near interfaces.

2. Methodology

The authors propose a Robust Deep First-Order System Least Squares (FOSLS) framework that integrates reduced-order modeling with deep learning.

A. Theoretical Framework: Weighted FOSLS

Instead of the standard second-order PDE, the problem is reformulated as a first-order system:
$$\text{div } \mathbf{q} = f, \quad \mathbf{q} + \kappa \nabla u = 0$$
where $u$ is the primal variable and $\mathbf{q}$ is the flux.

• Robust Energy Norm: The authors define a weighted norm for the error $(u, \mathbf{q})$ that incorporates the energy-norm Poincaré constant ($C_P^\kappa$).
  $$|||(u, \mathbf{q})|||_\kappa^2 = \|u\|_{H^1_{0,\kappa}}^2 + \|\mathbf{q}\|_{(L^2_\kappa)^d}^2 + C_P^\kappa{}^2 \|\text{div } \mathbf{q}\|_{L^2}^2$$
• Weighted Loss Functional: A corresponding loss functional is constructed:
  $$\mathcal{L}(u, \mathbf{q}) = \|\kappa^{-1/2}\mathbf{q} + \kappa^{1/2}\nabla u\|_{(L^2)^d}^2 + 2(C_P^\kappa)^2 \|\text{div } \mathbf{q} - f\|_{L^2}^2$$
• Key Theoretical Result: The authors prove that the equivalence constants between the loss and the true error are independent of the material parameter $\kappa$. This ensures that minimizing the loss guarantees error reduction even in high-contrast scenarios.

B. Numerical Strategy: Hybrid Solver-in-the-Loop

The method employs a "solver-in-the-loop" strategy combining linear algebra and gradient descent:

1. Neural Parametrization: A neural network generates a low-dimensional subspace spanned by basis functions $\{\phi_i, \tau_i\}$.
2. Linear Solver: For fixed network parameters $\theta$, the optimal coefficients are computed via a linear least-squares solver (with Tikhonov regularization and diagonal scaling for stability).
3. Nonlinear Update: The network parameters $\theta$ are updated using Stochastic Gradient Descent (SGD/Adam) to minimize the loss over the subspace.
4. Poincaré Constant Estimation: The constant $C_P^\kappa$ is estimated dynamically by solving a small generalized eigenvalue problem on the current discrete subspace.

C. Activation Function Selection

To mitigate the quasi-Gibbs phenomenon (oscillations near discontinuities), the authors select the Rectified Quadratic Unit (ReQU) activation function:
$$\text{ReQU}(x) = (\max\{0, x\})^2$$

• Rationale: ReQU networks can represent discontinuities in derivatives without the high-frequency oscillations typical of smooth activations (like tanh) or the distributional derivative issues of ReLU.

D. Passive Variance Reduction

A critical theoretical contribution is the proof of Passive Variance Reduction.

• In standard VPINNs/Deep Ritz, gradient variance remains high even as the solution converges.
• In the proposed FOSLS formulation, the variance of the stochastic gradient is proportional to the loss value itself. As the solution approaches the exact one (loss $\to$ 0), the gradient variance naturally vanishes, accelerating convergence and allowing for stable training with fewer quadrature points.

3. Key Contributions

1. Robust Equivalence: Proved that the FOSLS loss functional is equivalent to the physical energy norm with constants independent of material coefficients, solving the high-contrast instability issue.
2. Passive Variance Reduction: Demonstrated theoretically and numerically that the proposed loss landscape inherently reduces gradient variance as training progresses, a property absent in Deep Ritz and VPINNs.
3. ReQU for Discontinuities: Showed that using ReQU activation functions effectively suppresses quasi-Gibbs oscillations at material interfaces without requiring domain decomposition or adaptive meshing.
4. Hybrid Solver Architecture: Developed a stable, efficient training loop that leverages exact linear least-squares solvers for coefficients while using gradient descent to optimize the approximation space.

4. Numerical Results

Experiments were conducted in 1D and 2D settings with discontinuous coefficients and high contrast ratios ($10^{-6}$ to $10^6$).

• Stability vs. Quadrature Points: Unlike Deep Ritz, which required thousands of quadrature points to avoid overflow and instability, the FOSLS method remained stable with significantly fewer points (e.g., $N=50$ to $600$).
• Robustness to Contrast: In high-contrast tests, the standard FOSLS formulation showed a decoupling between loss and error. The proposed robust formulation maintained a linear correlation, accurately predicting error reduction.
• Quasi-Gibbs Mitigation: Comparisons between ReQU and tanh networks on interface problems showed that ReQU networks produced negligible oscillations near the interface, whereas tanh networks exhibited significant overshoots.
• Accuracy: The method achieved relative errors below 2% in the energy norm for both the primal variable and flux, even when the exact solution had discontinuous gradients or fluxes.

5. Significance

This work provides a mathematically rigorous and numerically stable framework for solving transmission problems using deep learning.

• It bridges the gap between classical finite element theory (via least-squares and energy norms) and modern deep learning.
• It solves the "variance bottleneck" in variational neural networks, making them viable for problems with high material contrast where standard methods fail.
• It offers a practical solution to the Gibbs phenomenon in neural PDE solvers through activation function selection, avoiding complex domain decomposition strategies.

The proposed method is particularly significant for applications in composite materials, geophysics, and heat transfer, where material properties change abruptly and high accuracy near interfaces is critical.