💻 computer science

Revisiting Neural Processes via Fourier Transform and Volterra Series

This paper proposes a new class of translation-equivariant Neural Processes that leverage Volterra series for analytical transparency and introduce Set Fourier Convolutions to efficiently model irregularly sampled data with global receptive fields and linear scaling.

Original authors: Peiman Mohseni, Nick Duffield, Raymond K. W. Wong

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Peiman Mohseni, Nick Duffield, Raymond K. W. Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the Unseen

Imagine you are a weather forecaster. You have a few scattered weather stations (some in cities, some in the middle of nowhere) reporting temperature and wind. Your goal is to predict the weather for every other point on the map, even where there are no sensors.

In the world of machine learning, this is called Neural Processes (NPs). These are smart models that learn from a few messy, irregular data points to guess the shape of the whole invisible curve (the "latent function") that generated them.

However, existing models have two big problems:

The "Grid" Problem: Some models try to force all data onto a perfect, neat grid (like graph paper). If your data is scattered or the map is huge, this grid becomes too big to handle, or you lose detail.
The "All-to-All" Problem: Other models (like Transformers) look at every single data point and compare it to every other point to find patterns. This is incredibly accurate but gets slow and expensive very quickly as you add more data (like trying to introduce every person at a party to every other person).

This paper proposes a new way to solve both problems at once.

The Two Main Ideas

The authors introduce two mathematical tools to fix these issues: Volterra Series and Fourier Transforms.

1. The Volterra Series: The "Recipe" for Complexity

The Problem: Current models are like black boxes. They stack layer after layer of math, and we don't really know why they work or what kind of patterns they are actually learning. It's hard to interpret.

The Solution: The authors use something called the Volterra Series.

The Analogy: Imagine you are baking a cake. A simple model might just say, "Mix flour and sugar." A complex model might say, "Mix flour, sugar, eggs, butter, vanilla, and bake at 350 degrees for 30 minutes, but if it's raining, add an extra egg."
The Volterra Series is like a master recipe book that breaks down any complex "mixing" process into a sum of simpler steps:
- Step 1: Just the ingredients (Linear).
- Step 2: How ingredients interact with each other (Quadratic).
- Step 3: How three ingredients interact (Cubic), and so on.

By using this, the authors can describe their model's behavior clearly (it's "analytically transparent") rather than just guessing. They can prove exactly how the model combines data points.

2. Set Fourier Convolutions (SFConvs): The "Radio Tuner"

The Problem:

Grid models (like ConvCNPs) are fast but need data to be on a perfect grid. If you have data from a jagged coastline or a 3D cloud, you have to force it into a box, which wastes memory and loses detail.
Attention models (like Transformers) work on messy data but are slow because they compare every point to every other point.

The Solution: The authors use Fourier Transforms (SFConvs).

The Analogy: Imagine you are listening to a radio.
- Spatial Domain (The old way): You look at the sound wave as it moves through time. To understand the whole song, you have to analyze every single wiggle in the wave. This is hard if the wave is messy.
- Frequency Domain (The new way): You tune the radio to specific frequencies (notes). Instead of looking at the whole messy wave, you just look at the "Low Bass," "Mid-Treble," and "High-Pitch" knobs.
Why it works: Most natural signals (like weather, images, or fluid flow) have most of their important information in the "low notes" (low frequencies).
The Magic: By working directly with these "notes" (frequencies) instead of the messy "wave" (spatial points), the model:
1. Skips the Grid: It doesn't need to force data onto a grid. It can handle scattered points directly.
2. Sees Everything at Once: In the frequency world, one "note" connects to the whole signal. This gives the model a "global view" (it can see how a storm in London affects weather in Paris) without having to compare every single point individually.
3. Stays Fast: It scales linearly (adding more data adds a little bit of work), unlike the slow quadratic scaling of attention models.

The New Models: SFConvCNPs and SFVConvCNPs

The authors built two new models based on these ideas:

SFConvCNPs: These models use the "Radio Tuner" (Fourier) approach. They stack layers of these frequency-based operations. They are fast, handle messy data, and see the "big picture."
SFVConvCNPs: These models combine the "Radio Tuner" with the "Recipe Book" (Volterra). They use the Fourier method to handle the data efficiently, but they structure the math using the Volterra series to ensure the model is interpretable and captures complex non-linear interactions.

What Did They Find? (The Results)

The authors tested these new models against the best existing models (like standard Neural Processes, Transformers, and Convolutional Neural Processes) on several challenges:

Synthetic Math Tasks: They tried to predict jagged waves (sawtooth waves) and smooth curves. The new models were better at predicting the sharp, jagged parts where other models failed.
Predator-Prey Dynamics: They simulated foxes and rabbits. The new models predicted the population cycles accurately, even when moving from simulated data to real historical data (Hudson Bay hare-lynx records).
Image Completion: They tried to "fill in the blanks" of missing parts of images (like CIFAR-10). The new models did a great job reconstructing the missing pixels.
Fluid Dynamics & Climate: They modeled complex swirling fluids (Kolmogorov flow) and real-world climate data (ERA5 temperature).
- Key Win: When they tested the models on a region of Europe they hadn't seen during training (a "shifted" map), the new models (which respect translation symmetry) performed perfectly. The old models, which relied on absolute positions, failed completely when the map moved.

Summary

Think of this paper as upgrading the engine of a car.

Old cars either had great suspension (good for smooth roads/grids) but couldn't handle off-road terrain, or they had great off-road tires (attention) but were too heavy and slow.
This paper introduces a new suspension system (Volterra) that explains exactly how the car handles bumps, and new tires (Fourier) that can drive on any terrain (scattered data) without getting stuck, while remaining fast and fuel-efficient.

The result is a model that is fast, accurate, interpretable (we know how it works), and robust (it works even when the data moves to a new location).

Technical Summary: Revisiting Neural Processes via Fourier Transform and Volterra Series

Problem Statement

Modeling unknown latent functions from finite, irregularly sampled measurements is a persistent challenge in scientific and engineering domains, such as turbulence modeling and climate science. While Neural Processes (NPs) offer a probabilistic framework for learning distributions over functions, existing translation-equivariant variants face two primary limitations:

Lack of Interpretability: Current architectures typically stack generic components with non-linearities. While empirically effective, this obscures the induced function class, limiting principled analysis and interpretability.
Scalability Trade-offs: Existing translation-equivariant NPs suffer from complementary scalability bottlenecks. Convolutional CNPs (ConvCNPs) rely on dense uniform grids to leverage convolutional networks, which becomes computationally prohibitive in high-dimensional or large spatial domains and restricts receptive fields to local neighborhoods. Conversely, attention-based methods (e.g., TE-TNPs) operate directly on unstructured inputs and model global interactions but scale quadratically with the number of observations, rendering them impractical for large context sets.

Methodology

The authors propose a framework that reconciles analytical transparency with practical efficiency through two core technical innovations: the Volterra series representation and Set Fourier Convolutions (SFConvs).

1. Volterra Representation of Equivariant Operators

The paper extends the convolutional deep set framework by characterizing continuous translation-equivariant operators using the theory of Volterra series.

Analytical Characterization: Analogous to a Taylor expansion, a Volterra series expresses any continuous translation-equivariant operator as an infinite sum of multi-linear convolution operators of increasing order. This provides an explicit, constructive characterization of the operator $\rho_d$ , offering analytical transparency absent in standard deep learning constructions.
Efficient Approximation: Direct evaluation of high-order convolutions is computationally infeasible. The authors leverage the multilinearity of Volterra operators and Fubini's theorem to reorganize higher-order terms into a nested (cascade) form. This reduces the evaluation of a truncated Volterra expansion to computing only first- and second-order convolutions.
Low-Rank Decomposition: To address the computational bottleneck of second-order convolutions (which involve quadratic tensor products), the authors impose a low-rank structure on the quadratic kernel via Hilbert–Schmidt spectral decomposition. This approximates the second-order term as a sum of products of first-order convolutions, parameterized and learned directly without enforcing strict orthonormality constraints.

2. Set Fourier Convolutions (SFConvs)

To overcome the discretization and locality limitations of grid-based methods, the authors introduce SFConvs, a frequency-domain parameterization of convolutional deep sets.

Direct Frequency Parameterization: Instead of discretizing inputs onto a spatial grid, SFConvs operate directly on the functional embedding of the context set in the Fourier domain. By the convolution theorem, convolution in the spatial domain corresponds to pointwise multiplication in the frequency domain.
Handling Irregular Inputs: The functional embedding $\rho_e[D_c]$ admits a closed-form Fourier transform when using Gaussian kernels. This allows the computation of spectral coefficients at arbitrary frequencies without spatial discretization, avoiding the "grid mismatch" issues where changing grid resolution alters the physical frequencies associated with learned weights.
Global Receptive Fields & Linear Scaling: Frequency-domain multiplication induces approximately global receptive fields, enabling the capture of long-range dependencies. Crucially, because the method operates on a fixed, compact frequency grid $\Xi$ (where $|\Xi| \ll |G|$ , the size of a dense spatial grid), the computational complexity scales linearly with the number of observations, $O(|\Xi|(|D_c| + |D_q|))$ .

3. Proposed Models

Building on these components, the authors propose two classes of Conditional Neural Processes (CNPs):

SFConvCNPs: Constructed by stacking SFConv blocks interleaved with pointwise non-linearities.
SFVConvCNPs: Constructed within the Volterra framework, integrating the Volterra formulation with SFConvs to explicitly approximate higher-order interactions.

Key Contributions

Principled Characterization: The paper provides a principled characterization of continuous translation-equivariant operators as sums of higher-order convolutions via the Volterra series, offering an analytically transparent alternative to opaque deep learning stacks.
Set Fourier Convolutions: The introduction of SFConvs alleviates the scalability limitations of grid-based methods by operating directly in the Fourier domain on irregular inputs, inducing global receptive fields while retaining linear complexity.
New Model Architectures: The proposal of SFConvCNPs and SFVConvCNPs, which demonstrate strong empirical performance across synthetic and real-world benchmarks, outperforming or matching state-of-the-art baselines.

Experimental Results

The authors evaluated their methods on five regression benchmarks with input dimensionalities $d_X \in \{1, 2, 3\}$ :

Synthetic 1D Regression: On tasks involving RBF, Matérn, periodic, sawtooth, and square-wave processes, SFConvCNPs and SFVConvCNPs consistently outperformed or matched competing methods (including CNPs, AttnCNPs, TNP, TE-TNP, and ConvCNPs). Notably, attention-based models collapsed on the sawtooth task (a high-frequency challenge), whereas the proposed convolution-based models succeeded.
Predator–Prey Regression: On simulated Lotka–Volterra trajectories and real-world Hudson Bay hare–lynx data, the SFConv-based models achieved competitive or superior predictive log-likelihoods and CRPS scores compared to baselines.
2D Image Completion: On CIFAR-10 and SVHN datasets, SFConvCNPs achieved the best predictive log-likelihoods, edging out grid-based spectral baselines (SConvCNPs) and transformer-based models.
3D Spatiotemporal Regression:
- Kolmogorov Flow: SFConvCNPs achieved the highest log-likelihood (1.89) and lowest CRPS (0.03) on complex fluid dynamics tasks, outperforming grid-based ConvCNPs and non-convolutional baselines.
- ERA5 Climate: In a sim-to-real transfer task predicting air temperature over Europe, SFConvCNPs demonstrated superior generalization to disjoint geographic regions (West and North) compared to non-equivariant baselines, which suffered significant performance degradation. The equivariant models maintained performance, confirming their reliance on relative displacements rather than absolute positions.

Significance and Claims

The paper claims to present a principled, scalable framework for translation-equivariant conditional neural processes. Its significance lies in:

Bridging Theory and Practice: By using Volterra series, the authors make the representational structure of convolutional deep sets analytically transparent, moving beyond empirical heuristics to systematic operator-theoretic design.
Resolving Scalability Trade-offs: The SFConv approach resolves the conflict between the need for global interactions and the need for linear scalability. It avoids the exponential cost of spatial discretization in high dimensions while avoiding the quadratic cost of attention mechanisms.
Robust Generalization: The models demonstrate robust generalization to unseen spatial regions and irregular sampling patterns, a critical requirement for scientific applications like climate modeling and fluid dynamics.

The authors emphasize that their work introduces frequency-domain neural processes capable of efficient probabilistic inference on irregularly sampled spatiotemporal data, combining linear scalability with global interaction and interpretable operators.