Solver-Integrated Adversarial Attacking and Training of Neural Operators

Technical Summary: Solver-Integrated Adversarial Attacking and Training of Neural Operators

Problem Formulation
Neural operators (e.g., Fourier Neural Operators, DeepONets) serve as fast data-driven surrogates for numerical Partial Differential Equation (PDE) solvers, mapping input functions (e.g., initial conditions, coefficients) to solution functions. While effective, their generalizability and robustness in the operator-learning setting remain ill-defined compared to traditional adversarial robustness in classification.

A critical distinction in PDE operator learning is that the "ground truth" is not a fixed label but the output of a numerical solver $G$. In traditional regression or classification, adversarial attacks assume the ground truth remains constant under input perturbation. However, in PDE settings, perturbing the input function $x$ inherently changes the true solution $G(x)$. Consequently, a model $F_\theta$ is not expected to be robust in isolation (i.e., its output should not change); rather, the discrepancy between the model and the solver, defined as the error operator $E_\theta = F_\theta - G$, must be robust. Existing methods often fail to account for this dynamic ground truth, either treating the solver output as a fixed label or ignoring the solver's gradient during attack generation.

Methodology

1. Formalizing Generalization and Robustness
The authors define robustness and generalization through the lens of the model-solver error operator $E_\theta$:

• Generalization: Defined statically as the norm of the error operator at a fixed input: $L_{gen}(x) = \|F_\theta(x) - G(x)\|_Y$.
• Robustness: Defined dynamically as how the error operator's output amplifies under input perturbation. The paper proposes three local metrics:
  • Jacobian Operator Norm: The induced norm of the derivative of the error operator, $\|DE_\theta[x]\|$.
  • Finite Adversarial Attack Loss Increase: The maximum increase in error loss over a norm-bounded perturbation ball.
  • Jacobian-Error Metric: A computationally efficient proxy defined as $\|g_\theta(x)\|_X$, where $g_\theta(x) = DE_\theta[x]^* e_\theta[x]$. This metric combines the error field with the local Jacobian and aligns closely with finite-budget attack directions in the small-budget limit.

2. Solver-Integrated Adversarial Attacks
The paper identifies that standard attack objectives (e.g., maximizing model output change or comparing against a fixed ground truth) are insufficient because they do not reflect the true model-solver discrepancy under perturbation. The authors propose a hierarchy of attack objectives based on the degree of solver integration:

• $L_1$ (Model-only): Maximizes change in model output ($\|F_\theta(x+\delta) - F_\theta(x)\|$).
• $L_2$ (Fixed Ground Truth): Maximizes error against the clean solver output ($\|F_\theta(x+\delta) - G(x)\|$).
• $L_3$ (Full Solver-Integrated): Maximizes the error against the perturbed solver output ($\|F_\theta(x+\delta) - G(x+\delta)\|$).

The core contribution is the $L_3$ objective, which requires the numerical solver $G$ to be differentiable and integrated into the attack loop. This allows the attack to compute gradients through the solver ($DG$), ensuring the perturbation $\delta$ is optimized to maximize the divergence between the model and the actual solution of the perturbed input. The paper also details optimization strategies (e.g., "replace" vs. "add" updates) that adapt to the geometry of the loss landscape (broad for Burgers, narrow for Navier-Stokes).

3. Solver-Integrated Adversarial Training
Building on the $L_3$ attack, the authors propose an adversarial training framework where the training loop operates as a min-max problem:
$$\min_\theta \mathbb{E}_{x \sim \mu_X} \left[ \max_{\|\delta\| \le \epsilon} \|F_\theta(x+\delta) - G(x+\delta)\|^2 \right]$$
In this pipeline, the current model is frozen to generate adversarial samples by maximizing the $L_3$ loss. These samples, paired with their corresponding solver outputs (re-labeled by the solver), are then used to update the model. This process acts as a "teacher-student" distillation, where the numerical solver acts as the teacher, and the neural operator learns to be a faithful behavioral twin of the solver across perturbed inputs.

Key Contributions

• New Robustness Definition: A formalization of generalization and robustness in operator learning based on model-solver discrepancy rather than model-only stability.
• Solver-Integrated Attack Target: Identification of the $L_3$ loss (full solver integration) as the most accurate target for adversarial attacks in PDE settings, demonstrating that attacks ignoring the solver's response to perturbation ($L_1, L_2$) or using fixed labels are suboptimal.
• Adversarial Training Framework: A method for generating solver-relabelled adversarial batches online, improving both generalization to shifted distributions and robustness to perturbations without manual sample selection.
• Metric Analysis: Demonstration that the Jacobian-error metric is a practical, lightweight proxy for finite-budget attack performance and that standard spectral norms are insufficient for measuring PDE operator robustness.

Experimental Results
The framework was evaluated on three canonical PDE benchmarks: 1D Burgers, 2D Darcy Flow, and 2D Navier-Stokes.

• Attack Effectiveness: Attacks using the full $L_3$ objective consistently achieved significantly higher model-solver loss increases compared to $L_1$, $L_2$, physics-residual, or dictionary-based attacks. $L_3$ attacks also generated higher-frequency perturbations, exposing specific model-solver mismatches.
• Training Performance:
  • Generalization: Models trained with $L_3$ adversarial training showed superior performance on shifted input distributions (e.g., different correlation lengths, kernel parameters, or boundary conditions) compared to baseline models and those trained with random perturbations or non-integrated losses.
  • Robustness: $L_3$-trained models exhibited the smallest attack loss increases and the lowest Jacobian-error norms, indicating stronger robustness.
  • Efficiency: While solver integration increases computational cost (due to forward and backward passes through the solver), the method yields more informative samples and faster convergence to robust solutions compared to heuristic sampling methods.

Significance and Claims
The paper claims to provide a general framework for robust operator training that leverages the existing numerical solver as a "trusted oracle." By integrating the solver directly into the attack and training loops, the method automatically selects difficult samples and aligns the neural operator's behavior with the solver's physics, reducing the need for manual intervention or hand-crafted shifted datasets.

The authors acknowledge that the primary limitation is computational cost, particularly for time-dependent PDEs where differentiating through long solver rollouts requires significant GPU memory. They suggest future work could explore hierarchical approaches (e.g., implicit neural rollouts) to mitigate this cost. The paper does not claim to solve all robustness issues in scientific machine learning but establishes that solver integration is a necessary component for defining and achieving true robustness in the operator-learning setting.