Walsh-Hadamard Neural Operators for Solving PDEs with Discontinuous Coefficients

This paper introduces the Walsh-Hadamard Neural Operator (WHNO), a novel architecture utilizing Walsh-Hadamard transforms to effectively solve partial differential equations with discontinuous coefficients by overcoming the limitations of Fourier-based methods, and demonstrates that combining WHNO with Fourier Neural Operators in an ensemble yields significantly superior accuracy in capturing both sharp interfaces and smooth features.

The Gist

Imagine you are trying to teach a computer to predict how heat flows through a complex machine or how a shockwave moves through water. These are problems governed by Partial Differential Equations (PDEs). Usually, computers solve these by breaking the problem down into tiny pieces and calculating the answer step-by-step, which is slow.

Neural Operators are a new kind of AI designed to learn the "rules" of these equations so they can predict the answer instantly, like a super-fast shortcut.

However, there's a catch. Most of these AI shortcuts rely on a mathematical tool called the Fourier Transform. Think of the Fourier Transform like a set of smooth, wavy sine waves (like gentle ocean swells). These waves are great for smooth, continuous things, but they struggle when the problem involves sharp, sudden jumps—like a wall of ice suddenly appearing in a river, or a material that changes from wood to metal instantly.

When you try to describe a sharp square edge using only smooth waves, the waves get confused. They start "ringing" or vibrating wildly near the edge, creating errors. In math, this is called the Gibbs phenomenon. It's like trying to draw a perfect square using only a round brush; you'll always get fuzzy, wobbly corners.

The New Solution: The Walsh-Hadamard Neural Operator (WHNO)

The authors of this paper introduced a new AI model called WHNO. Instead of using smooth waves, they swapped the brush for a set of rectangular blocks.

• The Analogy: Imagine you are tiling a floor.
  • The old method (Fourier) tries to tile a square room using only curved, wavy tiles. To make a straight wall, you have to stack thousands of tiny, curved pieces, and they never quite line up perfectly.
  • The new method (WHNO) uses square tiles. If you need a straight wall or a sharp corner, you just place the square tiles right next to each other. It fits perfectly, with no wobbly edges.

Because many real-world problems involve sudden changes (like a rock layer in the ground or a sharp temperature jump), these "rectangular wave" tiles are much better at capturing the truth without the confusing vibrations.

The "Best of Both Worlds" Strategy

The researchers didn't just stop at the new method. They realized that while the "square tiles" (WHNO) are great for sharp edges, the "smooth waves" (Fourier) are still very good at describing the smooth, gentle parts of the problem in between the edges.

So, they created a Team-Up (Ensemble).

• They trained two separate AIs: one with the square tiles (WHNO) and one with the smooth waves (FNO).
• They then mixed their predictions together, like blending two colors of paint.
• They used a smart testing process (cross-validation) to find the perfect "mixing ratio" for each specific problem.

The Result:
In every single test they ran—whether it was heat moving through weirdly shaped materials or shockwaves moving through fluid—the mixed team performed better than either AI working alone.

• Sometimes the mix was 57% square tiles and 43% smooth waves.
• Other times it was 65% square tiles and 35% smooth waves.

Even in cases where the "square tile" AI wasn't clearly the winner on its own, adding a little bit of the "smooth wave" AI still made the final answer more accurate.

Key Takeaways from the Paper

1. The Tool Matters: Changing the mathematical "brush" from smooth waves to rectangular blocks significantly improved accuracy for problems with sharp jumps, without making the computer slower.
2. Teamwork Wins: Combining the two different approaches (the new rectangular one and the old smooth one) always produced the best results. The two methods cover each other's weaknesses.
3. No Magic, Just Math: The paper tested this on specific physics problems (heat conduction and fluid shockwaves). It did not claim this works for medical diagnoses or other unrelated fields, but rather that for these specific types of "sharp jump" physics problems, this new combination is the most accurate method tested so far.

In short, the paper says: If you have a problem with sharp edges, don't just use the old smooth-wave AI. Use a new block-based AI, or even better, let the block-based AI and the smooth-wave AI work together as a team.

Technical Summary

Technical Summary: Walsh-Hadamard Neural Operators for Solving PDEs with Discontinuous Coefficients

Problem Statement
Neural operators have emerged as powerful tools for learning solution operators of parametric partial differential equations (PDEs), offering discretization-invariant surrogates that generalize across resolutions. However, standard spectral methods, particularly the Fourier Neural Operator (FNO), face significant limitations when applied to PDEs involving discontinuous coefficients or initial conditions. The Fourier basis, composed of smooth trigonometric functions, suffers from the Gibbs phenomenon when representing sharp interfaces or step functions. This results in oscillatory artifacts, slow point-wise convergence, and error localization near material boundaries. These issues are critical in applications such as subsurface flow in porous media (abrupt permeability contrasts), thermal management in composites (sharp conductivity jumps), and fluid dynamics (shock waves).

Methodology
The authors introduce the Walsh-Hadamard Neural Operator (WHNO), a spectral neural operator architecture that replaces the Fourier transform with the Walsh-Hadamard transform (WHT).

• Spectral Basis: Unlike the global trigonometric basis of the FNO, the WHT utilizes an orthonormal basis of rectangular wave functions. These functions are piecewise constant, making them naturally suited to represent step discontinuities without oscillatory ringing. The WHT decomposes signals exactly for piecewise constant fields and allows for aggressive spectral compression.
• Architecture: The WHNO architecture mirrors the FNO structure but operates in the Walsh-Hadamard domain.
  • Input Encoding: Input fields (e.g., conductivity or initial velocity) are concatenated with a constant channel.
  • Spectral Layers: The core of the network consists of spectral layers that transform inputs via the Fast Walsh-Hadamard Transform (FWHT). Learnable weights are applied to low-sequency coefficients (analogous to low-frequency modes in FNO) to capture global dependencies. The transform is then inverted back to the spatial domain.
  • Spatial Refinement: A dilated convolutional decoder processes the spectral features to resolve fine-scale features at discontinuities.
  • Complexity: The FWHT shares the $O(n \log n)$ computational complexity of the Fast Fourier Transform (FFT), ensuring WHNO maintains the same per-pass efficiency as FNO.
• Ensemble Strategy: Recognizing that Walsh-Hadamard bases excel at sharp interfaces while Fourier bases remain efficient for smooth regions, the authors propose a weighted ensemble of WHNO and FNO. The ensemble prediction is defined as $u_{ensemble} = w \cdot u_{WHNO} + (1-w) \cdot u_{FNO}$, where the weight $w^* \in [0, 1]$ is determined via five-fold cross-validation to minimize Mean Squared Error (MSE) on a validation set.

Key Contributions

1. Architecture Development: The formulation of the WHNO, which integrates Walsh-Hadamard spectral processing with learnable weights acting on low-sequency coefficients.
2. Validation on Discontinuous PDEs: Rigorous testing on two distinct problems:
   • Heat Conduction: Quasi-steady diffusion with discontinuous thermal conductivity (rectangular inclusions).
   • 2D Burgers Equation: Nonlinear advection-diffusion with discontinuous initial conditions (square blocks).
3. Controlled Comparisons: A head-to-head evaluation against FNO baselines with matched parameter counts (1,555,153 parameters) and identical training protocols.
4. Ensemble Framework: Demonstration that a simple, cross-validated scalar weight combining WHNO and FNO yields strictly superior performance to either model alone across all tested configurations.

Results
The study evaluated performance across seven configurations: four heat conduction geometries (axis-aligned, rotated, disks, Voronoi) and three Burgers initial condition families (block, smooth sinusoidal, oblique fronts).

• Single-Model Performance:

  • On the Heat Conduction problem, WHNO achieved lower Mean Absolute Error (MAE) and lower $H^1$ (gradient-MSE) error compared to FNO. Specifically, WHNO reduced MAE to $0.0153 \pm 0.00097$ versus FNO's $0.01663 \pm 0.00183$.
  • On the Burgers Equation, WHNO similarly outperformed FNO in MAE ($0.00642$ vs. $0.00843$) and $H^1$ error.
  • Error analysis revealed that FNO errors concentrated at material interfaces (Gibbs phenomenon), whereas WHNO errors were more uniformly distributed.
  • In configurations where discontinuities were not aligned with the Walsh basis (e.g., rotated rectangles, oblique fronts), the single-model advantage of WHNO diminished or reversed in $H^1$ metrics, though it often retained an edge in MAE.

• Ensemble Performance:

  • The cross-validated ensemble consistently outperformed both single models.
  • Heat Conduction: The optimal weight was $w^* = 0.572 \pm 0.016$. The ensemble achieved a test MSE of $3.65 \times 10^{-4}$, strictly lower than WHNO-only ($4.71 \times 10^{-4}$) and FNO-only ($5.66 \times 10^{-4}$).
  • Burgers Equation: The optimal weight was $w^* = 0.648 \pm 0.020$. The ensemble MSE was $6.6 \times 10^{-5}$, improving upon WHNO-only ($9.4 \times 10^{-5}$) and FNO-only ($1.59 \times 10^{-4}$).
  • Crucially, the ensemble achieved lower $H^1$ error than both baselines even in configurations where WHNO alone did not unambiguously beat FNO (e.g., heat with disk inclusions, Burgers with oblique fronts).

Significance and Claims
The paper claims two primary principles derived from these findings:

1. Spectral Basis Matters: At matched parameter counts, replacing the Fourier basis with the Walsh-Hadamard basis reduces error metrics (MAE and $H^1$) for PDEs with discontinuous structures, without increasing computational cost per forward pass.
2. Complementarity of Bases: The Walsh-Hadamard and Fourier representations capture partially complementary features of the solution field (sharp interfaces vs. smooth variations). Combining them via a single cross-validated scalar weight provides a robust method that strictly improves upon the Fourier baseline across all tested problem variants.

The authors position the WHNO+FNO ensemble as a practical, low-risk strategy for practitioners already using Fourier-based baselines to achieve measurable accuracy gains on problems with discontinuous coefficients or initial conditions, provided a validation set is available for weight tuning. The work acknowledges limitations, including the requirement for dyadic (power-of-two) grids and the sensitivity of the Walsh advantage to the alignment between the basis and the geometry of the discontinuities.