💻 computer science

Fourier Feature Pyramids for Physics-Informed Neural Networks

The paper introduces "beignet," a novel neural field architecture that replaces random Fourier features with a trainable multi-resolution Fourier feature pyramid to solve partial differential equations more accurately and efficiently than existing Physics-Informed Neural Networks by enabling spectral derivative computation, scalable parameterization, and stable optimization.

Original authors: Brandon Zhao, Yixuan Wang, Jonathan T. Barron, Katherine L. Bouman, Dor Verbin, Pratul P. Srinivasan

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Brandon Zhao, Yixuan Wang, Jonathan T. Barron, Katherine L. Bouman, Dor Verbin, Pratul P. Srinivasan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to predict how a fluid (like water or air) moves, or how heat spreads through a metal rod. These movements are governed by complex mathematical rules called Partial Differential Equations (PDEs).

For a long time, scientists have used a tool called Physics-Informed Neural Networks (PINNs) to solve these rules. Think of a standard PINN as a very smart, but slightly clumsy, student trying to learn a song. It listens to the notes (the math rules) and tries to sing them back. However, this student has a bad habit: they are great at learning the slow, low-pitched hums of the song, but they struggle terribly with the fast, high-pitched trills. In math terms, this is called "spectral bias." They learn the smooth parts of the solution quickly but miss the sharp, detailed edges.

To fix this, previous methods tried to give the student a "cheat sheet" of random high-pitched notes (called Random Fourier Features) to help them hear the fast parts. But this cheat sheet was messy. It was like giving the student a bag of random instruments; sometimes it helped, but often it made the song sound chaotic or required the student to memorize thousands of extra notes just to get the pitch right, which was slow and inefficient.

Enter "Beignet"

The authors of this paper introduce a new architecture they whimsically named Beignet (short for Bandlimited Embedding with Interpolated Grid Network).

Instead of a messy bag of random notes, Beignet gives the student a perfectly organized, multi-layered music sheet.

The Creative Analogy: The "Zooming Map"

Imagine you are trying to draw a detailed map of a city.

Old Method (Standard PINN): You try to draw the whole city on one giant piece of paper at once. If you zoom in to draw the tiny details of a single street, the whole map gets blurry and messy. If you zoom out to see the whole city, you lose the street details.
The "Beignet" Method: Beignet uses a pyramid of maps.
- Level 1 (The Base): A low-resolution map showing the whole city and major highways.
- Level 2: A medium map showing neighborhoods.
- Level 3: A high-resolution map showing individual streets.
- Level 4: A super-detailed map showing house numbers.

When the computer needs to know what's happening at a specific spot, it doesn't just look at one map. It looks at all the levels at once. It combines the big picture from the bottom layers with the fine details from the top layers.

How It Works (The Magic Tricks)

The paper claims Beignet is better than previous methods for three main reasons:

It Knows Exactly How "Zoomed In" It Is:
Unlike the old random cheat sheets, Beignet's "map pyramid" has a strict limit on how detailed it can get. The researchers can tell the computer, "Stop zooming in past this point." This prevents the computer from getting confused by noise or trying to learn impossible details, which makes the learning process much more stable.
It Does Math Super Fast:
Because the maps are arranged in a perfect grid (like a chessboard), Beignet can use a mathematical shortcut called the Fast Fourier Transform (FFT). Think of this as a magic calculator that can instantly figure out the slope and speed of the fluid at every point on the grid without having to do the hard math for every single dot individually. This makes the computer run much faster and use less memory.
It Adapts to the Problem:
Some physics problems need lots of high-frequency details (like a sharp shockwave), while others need smooth, low-frequency data. Beignet lets the researchers adjust the "pyramid" to fit the specific problem.
- Analogy: If you are solving a problem with sharp edges, you give the pyramid more high-resolution layers. If the problem is smooth, you give it fewer layers. This makes the solution more accurate without wasting computer power.

What Did They Prove?

The authors tested Beignet on four different physics problems (like how waves move or how chemicals react) and compared it to the best existing methods.

The Result: Beignet found much more accurate solutions using fewer computer parameters (less memory and brainpower) than the competition.
The "Impossible" Test: They also tested it on a problem where the solution blows up into a singularity (a point of infinite density). Usually, standard computers give up on this or need extremely expensive, slow super-computers to get close. Beignet, using a standard, fast optimizer (called Adam), managed to get the answer almost perfectly accurate—something previously thought impossible without expensive tools.

Summary

In short, Beignet is a smarter way to teach computers to solve physics equations. Instead of guessing randomly or trying to memorize everything at once, it uses a structured, multi-scale system (like a set of zooming maps) that is fast, stable, and highly accurate. It allows computers to see both the forest and the trees simultaneously, solving complex physical puzzles with unprecedented precision.

Technical Summary: Fourier Feature Pyramids for Physics-Informed Neural Networks

Problem Statement

Physics-Informed Neural Networks (PINNs) offer a flexible framework for solving partial differential equations (PDEs) by parameterizing solutions as coordinate-based neural networks. However, standard PINNs face two primary challenges:

Spectral Bias: Standard gradient descent optimization causes neural networks to learn low-frequency components of a target function significantly faster than high-frequency components. This hinders the accurate resolution of sharp interfaces, localized waves, and fine-scale spatial structures common in PDE solutions.
Scalability and Efficiency: Existing strategies to mitigate spectral bias, such as increasing the bandwidth of random Fourier features (RFFs) or scaling up the size of the underlying Multi-Layer Perceptron (MLP), introduce significant computational and optimization difficulties. Increasing MLP size drastically raises the cost of evaluating PDE derivatives via automatic differentiation, often necessitating memory-prohibitive second-order optimization methods to achieve high numerical accuracy. Furthermore, RFFs are not strictly bandlimited, and increasing their bandwidth can introduce unstable high-frequency components that brittle the optimization process.

Methodology: Beignet

The authors propose beignet (Bandlimited Embedding with Interpolated Grid Network), a neural field architecture that replaces the random Fourier feature embedding used in standard PINNs with a trainable multi-resolution Fourier feature pyramid.

Core Architecture

Multi-Resolution Fourier Pyramid: Instead of random frequencies, beignet utilizes a set of periodic feature grids at coarse-to-fine spatial resolutions. Each level $\ell$ of the pyramid stores a feature field at a prescribed grid resolution $N^{(\ell)}$ .
Fourier Interpolation: To query the network at a continuous coordinate $x$ , the model performs Fourier interpolation at each level of the pyramid to extract a feature vector. These vectors are concatenated and passed to a compact MLP decoder.
Bandlimit Control: The maximum resolution of the finest grid explicitly determines the maximum spatial frequency (bandlimit) the network can represent. This allows for direct control over the spectral content, preventing the introduction of uncontrolled high-frequency noise.
Temporal Handling: For time-dependent PDEs, the time axis is divided into bins. The model stores a set of learned temporal feature slices (pyramids) for each bin and linearly interpolates between them based on the specific time coordinate. The raw time coordinate is also passed to the decoder to capture smooth trends.

Efficient Derivative Computation

A key innovation of beignet is the efficient computation of spatial derivatives required for PDE residuals:

Spectral Differentiation: Derivatives of the feature grids are computed spectrally in the Fourier domain by multiplying Fourier coefficients by wavenumbers (using the Fast Fourier Transform, FFT).
Chain Rule Composition: These spectrally computed feature derivatives are then propagated through the MLP decoder using the chain rule (via Faà di Bruno's formula for higher-order derivatives).
Benefit: This approach decouples the cost of derivative computation from the MLP size, allowing for efficient evaluation of high-order derivatives without the memory overhead of dense derivative tensors.

Regularization

To prevent aliasing and overfitting to fixed collocation grids, the authors employ stochastic grid shifts. During training, the structured residual grid is randomly offset by a phase ramp in Fourier space, ensuring the network learns accurate solutions between grid points rather than just at the grid locations.

Key Contributions

Architectural Innovation: The introduction of a learnable, multi-resolution Fourier feature pyramid that replaces random Fourier features, offering explicit control over spatial bandwidth and frequency-band redundancy.
Efficient Differentiation: A method for computing spatial derivatives that combines spectral differentiation (FFT) with automatic differentiation through the decoder, enabling efficient evaluation of PDE residuals.
Parameter-Efficient Scaling: Demonstrating that scaling the capacity of the Fourier feature pyramid (rather than the MLP decoder) is a more efficient strategy for improving accuracy, particularly for problems requiring high-frequency resolution.
Stable Optimization: Showing that explicit bandlimiting and stochastic grid shifts lead to more stable optimization, enabling first-order optimizers (like Adam) to achieve near-machine precision on difficult problems where standard MLPs plateau.

Experimental Results

The authors evaluate beignet on two distinct regimes:

1. Time-Dependent PDE Benchmarks (JAX-PI Suite)

Beignet was tested on four time-dependent PDEs: Allen–Cahn, Korteweg–de Vries (KdV), Complex Ginzburg–Landau, and Gray–Scott.

Performance: Beignet achieved substantially lower relative $L_2$ errors compared to state-of-the-art baselines (JAX-PI and PirateNet) across all four tasks.
Efficiency: On the Gray–Scott and KdV problems, beignet reduced errors by more than an order of magnitude while using significantly fewer parameters than the baselines.
Spectral Allocation: The experiments revealed that different PDEs benefit from different spectral allocations. For example, the Allen–Cahn equation (with sharp interfaces) benefited from a wider pyramid bandwidth, whereas the KdV equation (with a third-order derivative) performed best with a compact, lower-bandwidth pyramid to avoid optimization stiffness caused by high-frequency error amplification.

2. Self-Similar Inviscid Burgers Blowup

The authors evaluated beignet on a self-similar singularity formation problem, a critical test for understanding PDE singularities (related to the Navier–Stokes existence problem).

Precision: Using only the first-order Adam optimizer, beignet drove the PDE residual to near-machine precision ( $\sim 10^{-19}$ ).
Comparison: In contrast, a standard coordinate-based MLP trained with Adam plateaued at much higher residuals ( $\sim 10^{-7}$ ). Achieving similar precision with the MLP baseline required computationally expensive second-order optimizers (BFGS) and quadratic memory scaling.
Implication: This demonstrates that the feature representation itself is a critical factor in enabling high-precision optimization without relying on expensive second-order methods.

Significance and Claims

The paper claims that beignet represents a significant step forward in scaling PINNs to high-accuracy regimes. By treating the feature embedding as a spectral allocation problem, the architecture allows for:

Direct Control: Explicit management of the representation's bandlimit, which stabilizes optimization for difficult PDEs.
Efficiency: Achieving higher accuracy with fewer parameters by scaling the feature pyramid rather than the decoder.
Robustness: Enabling first-order optimizers to reach precision levels previously attainable only with expensive second-order methods.

The authors position this work as a foundational improvement in neural PDE solvers, suggesting that carefully designing the feature representation is as crucial as the optimization algorithm for resolving fine-scale structures, extreme multiscale dynamics, and singularities in fluid dynamics and other physical systems. The paper does not claim to solve the Navier–Stokes existence problem but highlights the method's potential as a tool for investigating such fundamental open problems.