🤖 AI

Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEs

This paper introduces the Higher-Order Fourier Neural Operator (HO-FNO), a novel architecture that enhances the standard FNO by incorporating explicit n-linear mode mixing to better capture the structured interactions in nonlinear PDEs, thereby achieving superior performance and efficiency over existing spectral neural operators, transformers, and state-space models, particularly in highly nonlinear regimes.

Original authors: Alex Colagrande, Paul Caillon, Eva Feillet, Alexandre Allauzen

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alex Colagrande, Paul Caillon, Eva Feillet, Alexandre Allauzen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how a complex system, like a swirling fluid or a vibrating metal beam, will behave over time. In the world of physics, these systems are described by equations called Partial Differential Equations (PDEs). Solving these equations on a computer is usually like trying to count every single grain of sand on a beach to predict the tide—it's incredibly accurate but takes forever.

Recently, scientists started using AI to learn these patterns instead of solving the equations step-by-step. One popular AI method is called the Fourier Neural Operator (FNO). Think of FNO as a master chef who can taste a soup and instantly guess the recipe. It works by looking at the "flavors" (frequencies) of the data.

However, the standard FNO has a limitation. It assumes that these flavors mostly evolve on their own, like independent musicians playing a simple melody. But in the real world, especially with non-linear systems (where things get messy and chaotic), these flavors interact in complex ways. It's not just a solo; it's a jazz improvisation where the saxophone, drums, and bass are constantly reacting to each other, creating new sounds together.

The standard FNO struggles to hear these complex interactions because it only listens to the musicians individually.

The New Solution: HO-FNO

The authors of this paper introduce a new AI model called Higher-Order Fourier Neural Operator (HO-FNO).

Here is the core idea using a simple analogy:

The Old Way (FNO): Imagine a group of people in a room. The old AI model asks each person, "What are you thinking?" and updates their thought based only on their own answer. If two people need to have a conversation to solve a problem, the model misses it until the very end when it tries to guess the connection.
The New Way (HO-FNO): The new model realizes that in a chaotic room, people talk in groups. It explicitly allows groups of people (modes) to talk to each other during the thinking process. If Person A and Person B need to combine their ideas to influence Person C, HO-FNO sets up a direct line for that specific interaction.

Why This Matters

The paper claims that by building this "group conversation" directly into the AI's brain, it can solve complex physics problems much faster and more accurately.

One Layer vs. Sixteen: In their experiments, they created a tricky math problem (a Poisson equation with a complex, multi-layered source). They found that a single layer of their new HO-FNO could solve it better than a standard FNO with 16 layers. It's like a single genius solving a puzzle that would take a team of 16 average people hours to figure out.
Efficiency: Despite doing more complex "group thinking," the new model doesn't get slow. It keeps the speed of the old method because it still uses the same efficient math tricks (Fourier transforms) to handle the data.
Better than the Giants: They tested HO-FNO against other powerful AI models (like Transformers, which are famous for being smart but heavy). HO-FNO matched or beat them in accuracy while using less computer memory and time.

Where They Tested It

The authors didn't just talk about theory; they tested this on several real-world physics scenarios:

Fluid Dynamics: Simulating how air flows over an airplane wing or how water moves through a pipe.
Stress and Deformation: Predicting how metal bends or breaks under pressure.
Weather Patterns: Simulating water waves on a sphere (like the Earth).
Custom Math Tests: They created a specific test where the "source" of the problem was a product of multiple inputs (e.g., $Input_1 \times Input_2 \times Input_3$ ). This is exactly the kind of "group interaction" the new model was designed for, and it crushed the competition there.

The Bottom Line

The paper argues that many physical systems rely on specific types of interactions (like three waves crashing together to create a fourth). The old AI models had to guess these interactions by stacking many layers on top of each other. The new HO-FNO model builds these interactions directly into its design.

The result is a model that is smarter, faster, and more efficient at predicting how complex physical systems behave, particularly when those systems involve messy, non-linear interactions. The authors have even made their code public so others can try it out.

Technical Summary: Higher-Order Fourier Neural Operator (HO-FNO)

Problem Statement

Neural operators, particularly the Fourier Neural Operator (FNO), have established themselves as effective tools for learning mappings between function spaces, offering resolution-invariant performance and efficiency on periodic domains. However, standard FNO architectures rely on spectral convolutions that act diagonally on Fourier modes. This design assumes that modes evolve independently, which aligns well with linear, constant-coefficient Partial Differential Equations (PDEs) on periodic domains.

The paper identifies a critical limitation: nonlinear PDEs often exhibit structured interactions between Fourier modes governed by polynomial nonlinearities (e.g., the advective term in Navier-Stokes or polynomial forcing in Poisson equations). In these cases, the physical dynamics involve explicit mixing of modes (e.g., triadic interactions where $k_1 + k_2 = k_3$ ). Standard FNOs can only induce such mixing implicitly through pointwise nonlinearities applied after linear spectral layers. This forces the model to rely on weight sharing across different interaction orders and factors, creating a "tied" class of interactions that may not adequately capture the specific structure of the underlying PDE's nonlinearity.

Methodology: Higher-Order Spectral Convolution

To address this, the authors propose the Higher-Order Fourier Neural Operator (HO-FNO), which introduces an Explicit Mode Mixer via a Higher-Order Spectral Convolution.

Core Mechanism

The method extends the standard spectral convolution from diagonal modulation to explicit $n$ -linear mode mixing.

Physical Space Formulation: Instead of applying a linear operator followed by a pointwise nonlinearity, the HO-FNO layer computes an $m$ -linear product of transformed input fields in physical space:
$(H_\theta u)(x) = \int_\Omega \kappa_\theta(x-y) \mathcal{T}_M \left( (A_1 u)(A_2 u) \cdots (A_m u) \right)(y) dy$
Here, $A_i$ are learnable linear operators acting channel-wise, and $\mathcal{T}_M$ denotes Fourier truncation.
Fourier Space Formulation: By the convolution theorem, the pointwise product in physical space becomes a convolution in the Fourier domain. The update for a specific Fourier mode $k$ aggregates all $m$ -tuples of input modes $(k_1, \dots, k_m)$ such that their indices sum to $k$ :
$\widehat{(H_\theta v)}(k) = R(k) \sum_{k_1 + \dots + k_m = k} A_1 \hat{v}(k_1) \odot \dots \odot A_m \hat{v}(k_m)$
This structure explicitly parameterizes the $m$ -linear mixing of Fourier coefficients, mirroring the algebraic structure of polynomial nonlinearities in PDEs.

Architectural Details

Efficiency: The computational complexity remains $O(N \log N)$ per layer, matching standard FFT-based operators, avoiding the $O(N^m)$ complexity associated with higher-order attention mechanisms in the spatial domain.
Parameter Count: The additional parameters introduced by the $m$ linear maps ( $A_i$ ) are $mC^2$ (where $C$ is the channel dimension), which is negligible compared to the $MC^2$ parameters of the standard spectral convolution (where $M$ is the number of retained modes).
Generalization: The framework is extended to non-Euclidean geometries (e.g., spheres) by replacing the standard Fourier transform with the spectral decomposition of the Laplace-Beltrami operator (Spherical Harmonics), resulting in the Spherical HO-FNO (HO-SFNO).

Key Contributions

First Explicit Spectral Mixer for Nonlinear PDEs: The authors design the first spectral neural operator that explicitly models mode interactions corresponding to polynomial nonlinearities, moving beyond the diagonal modulation of standard FNOs.
Theoretical Alignment: The paper provides a theoretical argument (Appendix C) showing that pointwise nonlinearities induce a "tied" class of interactions where the same latent linear map is reused across factors. In contrast, HO-FNO realizes a broader class of interactions where each factor can have distinct linear transformations, allowing for a more precise approximation of the solution operator for degree- $n$ nonlinear PDEs.
Geometric Flexibility: The method is successfully adapted to spherical geometries by combining higher-order mixing with Spherical Harmonic convolutions.
Comprehensive Benchmarking: The authors evaluate HO-FNO across a wide range of standard benchmarks, including regular grids, structured meshes, and point clouds, comparing it against over 20 recent methods (Transformers, State-Space Models, and other Neural Operators).

Experimental Results

The experiments demonstrate that HO-FNO retains the efficiency of FNO while significantly improving accuracy, particularly in highly nonlinear regimes.

Polynomial-Source Poisson Equation: On a controlled dataset with polynomial forcing of degree $p$ , a single-layer HO-FNO of order $p$ consistently outperforms FNOs with up to 16 layers. For $p=5$ , the HO-FNO achieves a test loss two orders of magnitude lower than FNO.
Navier-Stokes Equations: HO-FNO outperforms standard FNO and other spectral baselines across three viscosity regimes ( $\nu \in \{10^{-3}, 10^{-4}, 10^{-5}\}$ ). Notably, it achieves state-of-the-art performance among frequency-based models, matching or exceeding Transformer and State-Space Model (SSM) baselines like LaMO and Transolver.
Standard Benchmarks: Across tasks including Darcy flow, Plasticity, Airfoil, Pipe, and Elasticity, HO-FNO consistently achieves the lowest relative $L_2$ $L_{2}$ error among frequency-based models and is competitive with the strongest Transformer/SSM baselines.
- Example: On the Plasticity dataset (involving non-polynomial, non-smooth nonlinearities), HO-FNO achieves an error of 0.0006, significantly outperforming the best frequency-based baseline (0.0010) and the best Transformer baseline (0.0011).
Spherical Shallow Water Equations (SWE): On the sphere, HO-SFNO reduces rollout error from $7.7 \times 10^{-1}$ (SFNO) to $7.0 \times 10^{-1}$ , demonstrating improved long-horizon stability.
Efficiency: HO-FNO adds negligible computational overhead compared to FNO (e.g., +1.8% training time on Navier-Stokes) while remaining substantially faster and more memory-efficient than Transformer and SSM baselines.

Significance and Claims

The paper claims that HO-FNO bridges the gap between the efficiency of spectral methods and the expressivity required for complex nonlinear dynamics.

Inductive Bias: The primary significance lies in the introduction of an inductive bias that aligns with the algebraic structure of polynomial nonlinearities in PDEs. By explicitly parameterizing mode mixing, the model captures the "triadic" (or $n$ -adic) interactions inherent in fluid dynamics and other physical systems without the computational cost of higher-order attention.
Depth Efficiency: The results suggest that HO-FNO is significantly more depth-efficient. A single layer of HO-FNO can outperform deep stacks of standard FNO layers, particularly when the order of the network matches the degree of the nonlinearity in the target PDE.
Generalization: The method maintains the resolution-invariance property of FNOs, proving that higher-order mixing does not compromise the ability to generalize across different discretizations.

The authors conclude that while HO-FNO is designed for polynomial nonlinearities, its success on non-polynomial tasks (like Plasticity) suggests that higher-order spectral interactions provide a robust and generalizable inductive bias for a broad class of nonlinear PDEs.