🤖 machine learning

Frequency Bias and OOD Generalization in Neural Operators under a Variable-Coefficient Wave Equation

This paper investigates the generalization capabilities of Fourier Neural Operators (FNO) and DeepONets under structured distribution shifts in a variable-coefficient wave equation, revealing that while both models handle coefficient smoothness shifts well, they exhibit distinct frequency biases where FNO struggles with unseen high frequencies and DeepONet degrades more mildly despite higher overall error.

Original authors: Runlong Xie, An Luo

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Runlong Xie, An Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach two different students how to predict how a wave will move through a long, wiggly rope. The rope isn't uniform; some parts are thick and heavy, while others are thin and light. This is what scientists call a "variable-coefficient wave equation."

The goal is to teach these students (which are actually computer programs called Neural Operators) to look at the starting shape of the rope and the varying thickness of the material, and then instantly tell you what the rope will look like a few seconds later. This is useful because calculating this with traditional math is slow and expensive, like doing a complex puzzle every single time.

The paper tests two specific "students":

FNO (Fourier Neural Operator): This student is like a musician who only knows how to play specific, pre-learned musical notes (frequencies). They are very good at playing songs made of the notes they practiced.
DeepONet: This student is like a painter who learns to mix colors and brushstrokes based on the shape of the canvas. They don't rely on a fixed list of notes but build the picture from scratch using a flexible system.

The Experiment: The "Surprise Test"

The researchers trained both students on a specific set of ropes: ones with gentle waves and smooth changes in thickness. Once the students mastered these, the researchers gave them a "surprise test" with two types of new, tricky ropes:

The "Rough" Rope (Smoothness Shift): A rope where the thickness changes very abruptly and chaotically.
The "Fast" Rope (Frequency Shift): A rope with very tiny, rapid, high-pitched vibrations that the students had never seen before.

What Happened?

1. The Smoothness Test (The Rough Rope)
Both students handled this pretty well. They could still predict the wave's movement accurately, even though the rope was bumpy. The "musician" (FNO) actually did slightly better here, keeping the prediction very clean.

2. The Frequency Test (The Fast Rope)
This is where the two students reacted very differently:

The Musician (FNO) Crashed: When faced with the rapid, high-pitched vibrations, the FNO student got confused. Because they were trained only on a specific set of "notes," they tried to force the new, fast vibrations into their limited vocabulary. The result was a prediction that looked nothing like the real wave—it was full of distortions and phase errors. It was like trying to describe a high-speed race car using only words for a slow bicycle; the description falls apart completely.
- The Twist: The researchers tried giving the musician more notes to memorize (increasing their capacity), but it didn't help. In fact, it made the prediction worse on the new ropes. This proved the problem wasn't that they didn't know enough notes; it was that their method of using notes was too rigid for new situations.
The Painter (DeepONet) Adapted: The DeepONet student also made mistakes, but they were much gentler. Instead of a total crash, the prediction was just a little "off" everywhere. The wave still looked like a wave, and the peaks and valleys were in roughly the right places. They didn't have a perfect answer, but they didn't hallucinate a completely wrong reality.
- Why? Because this student doesn't rely on a fixed list of notes. They build the solution piece by piece. When they see a new, fast vibration, they just adjust their brushstrokes slightly rather than trying to force it into a pre-defined slot.

The Big Takeaway

The paper concludes that being good at the training data doesn't guarantee you'll be good at the real world.

FNO is like a specialist who is incredibly fast and accurate at the specific tasks they practiced, but if the task changes even slightly (like a new frequency), they break down.
DeepONet is a bit slower and less perfect on the practice tasks, but they are more "robust." When the world throws them a curveball, they don't crash; they just give a slightly less accurate, but still recognizable, answer.

The Moral: In physics and engineering, if you want a model that won't fail when it encounters something new (like a new type of material or a faster wave), you have to look at how the model is built, not just how well it scores on a test. The "musician" approach is efficient but fragile; the "painter" approach is more flexible and safer for the unknown.

Technical Summary: Frequency Bias and OOD Generalization in Neural Operators under a Variable-Coefficient Wave Equation

1. Problem Statement

Neural operators are designed to learn mappings between input functions and the solutions of partial differential equations (PDEs), serving as surrogates for repeated numerical simulations. While architectures like the Fourier Neural Operator (FNO) and Deep Operator Network (DeepONet) have demonstrated strong in-distribution (ID) performance across various PDE tasks, their behavior under structured distribution shifts remains insufficiently understood.

This paper investigates a specific gap: whether strong ID performance guarantees stable extrapolation when the input frequency content or the medium structure (coefficient smoothness) differs from the training distribution. This is critical for real-world scientific applications where models must operate on physical systems that systematically differ from training data. The study focuses on a terminal-state operator learning task governed by a one-dimensional variable-coefficient wave equation, a fundamental model for wave propagation in heterogeneous media.

2. Methodology

2.1. Problem Formulation

The study defines an operator mapping $G: (u_0(x), c(x)) \mapsto u(x, T)$ , where:

$u_0(x)$ : Initial displacement field.
$c(x)$ : Spatially varying wave-speed coefficient (governing local propagation speed).
$u(x, T)$ : Wave solution at a fixed terminal time $T$ .
The equation used is the conservative variable-coefficient wave equation: $u_{tt} = \partial_x(c(x)^2 u_x)$ , with homogeneous Dirichlet boundary conditions and zero initial velocity.

2.2. Architectures Compared

The paper conducts a controlled empirical comparison between two representative architectures with distinct inductive biases:

Fourier Neural Operator (FNO): Utilizes spectral convolution in Fourier space. It lifts inputs to a latent space and updates them via Fourier layers that learn transformations among a fixed, truncated set of low-frequency modes. This imposes a bias toward smooth structures and assumes dominant solution features reside in low frequencies.
Deep Operator Network (DeepONet): Employs a branch-trunk decomposition. The branch network encodes input functions into coefficients, while the trunk network encodes spatial coordinates into basis functions. The output is an inner product of these, effectively learning a coordinate-dependent expansion without explicit fixed spectral truncation.

2.3. Experimental Design

To isolate architectural effects, the authors constructed structured Out-of-Distribution (OOD) settings while keeping other factors (model scale, pretraining) constant:

Frequency Shifts: Initial displacement fields ( $u_0$ ) containing higher-frequency components than those seen during training.
Smoothness Shifts: Coefficient fields ( $c(x)$ ) with varying levels of spatial variability (smooth, medium, rough), altering the heterogeneity of the medium.

Data was generated using a conservative finite-difference solver. Evaluation metrics included relative $L_2$ error, an energy-like diagnostic quantity, and spectral-error analysis to examine error distribution across Fourier modes.

3. Key Contributions

Task Formulation: Defined a terminal-state operator learning task based on a conservative variable-coefficient wave equation with controlled ID and structured OOD evaluation settings.
Controlled Comparison: Provided a direct, fair comparison between FNO and DeepONet under a unified experimental framework to assess architectural effects on generalization.
Mechanistic Insight: Combined spectral-error analysis with structured OOD testing to reveal how different operator representations interact with frequency structure, offering empirical insights into generalization mechanisms in physics-based PDE settings.

4. Results and Analysis

4.1. Performance Under Smoothness Shifts

Both models maintained stable performance when the smoothness of the coefficient field $c(x)$ changed. FNO achieved lower overall error in these scenarios.

4.2. Performance Under Frequency Shifts

A sharp divergence in behavior was observed when inputs contained unseen high-frequency components:

FNO: Exhibited a sharp increase in error. The degradation was not limited to the highest frequencies but manifested as elevated errors across a range of mid-to-high Fourier modes. Spectral analysis revealed that unseen high-frequency inputs disrupted the learned spectral relationships, causing errors to spread across multiple frequency bands. Increasing the number of retained Fourier modes (spectral capacity) did not mitigate this; in fact, OOD error increased with more modes, suggesting the issue is architectural representation rather than capacity.
DeepONet: Showed milder degradation despite having higher overall error in ID settings. The error increase was gradual and spatially distributed. Spectral analysis indicated a more uniform error distribution without the concentrated spikes seen in FNO. The model preserved the overall oscillatory structure (peak/trough locations) better, avoiding the severe phase shifts and local distortions observed in FNO.

4.3. Spatial-Spectral Correlation

The study found a consistent relationship between spatial and spectral errors. For FNO, localized spatial distortions under OOD-frequency conditions corresponded to multi-mode spectral deviations. For DeepONet, errors were more evenly distributed, lacking the structured accumulation across specific frequency bands.

5. Significance and Claims

The paper argues that architectural representation bias plays a central role in determining generalization under distribution shifts.

Limitation of ID Metrics: Strong in-distribution performance does not necessarily lead to stable extrapolation when input frequency structures change.
Architectural Bias: The sharp degradation of FNO under frequency shifts is attributed to its reliance on fixed spectral truncation and learned spectral relationships that fail to extrapolate beyond the training regime. Conversely, DeepONet's coordinate-dependent basis expansion allows for a smoother transition when encountering unseen frequencies, though it may not capture fine details as accurately as FNO within the training distribution.
Implication for Reliability: These findings underscore the need for structured OOD evaluation and spectral analysis to develop more reliable neural operators for physics-based simulations. The authors suggest that future architectures may need to explicitly adapt to the spectral properties of input functions (e.g., frequency-aware representations) to improve robustness.

The work concludes that understanding the failure modes of learned PDE surrogates is essential for their safe and reliable deployment in scientific and engineering applications where physical conditions may systematically differ from training data.