🤖 machine learning

Feature Interaction Modeling for Physics-Informed Neural Networks and Neural Operators

This paper proposes integrating feature interaction modules from factorization machines into physics-informed neural networks and neural operators to enhance the modeling of variable couplings, significantly improving accuracy for PDEs with strong cross-field dependencies, sharp gradients, or discontinuities while offering limited benefits for smooth operator learning tasks.

Original authors: Quan Gu, Hongxia Liu

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Quan Gu, Hongxia Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to predict how a physical system behaves, like how heat spreads through a metal plate, how a shockwave ripples through the air, or how traffic jams form on a highway. In the real world, these things are governed by complex mathematical rules called Partial Differential Equations (PDEs). For decades, solving these equations has been like trying to navigate a maze with a blindfold on; you need a detailed map (a grid) and very specific instructions for every single turn. But what if the computer could learn the rules of the maze just by looking at the destination? That's where a new generation of "smart" neural networks comes in. These networks, known as Physics-Informed Neural Networks (PINNs) and Neural Operators, are designed to learn the language of physics directly, without needing a pre-drawn map. They are incredibly powerful, but they sometimes struggle when the physics gets messy—like when variables interact in complicated ways or when the solution has sharp, jagged edges.

The big question researchers have been asking is: How can we make these networks better at understanding how different parts of a problem talk to each other? Think of it like a group project. If you have a team of students (variables like time, space, and temperature), a standard approach might ask each student to write their part independently and then glue the papers together. But in reality, students often need to collaborate, debate, and influence each other's work to get the best result. This paper explores a clever trick borrowed from a different field (recommender systems) to help these physics networks "collaborate" more effectively. By forcing the network to explicitly model how these variables interact—like how a change in time might change the effect of a change in space—the authors hope to build a smarter, more accurate predictor for the physical world.

The Paper's Big Idea: Teaching Networks to "Date" Their Variables

The authors of this paper, Quan Gu and Hongxia Liu, decided to give these physics-learning networks a dating app. Well, not a real one, but a mathematical mechanism called Factorization Machines (FMs). In the world of data science, FMs are famous for figuring out how different things pair up. For example, if you like "Sci-Fi" movies and "Action" movies, a standard system might just add those scores together. But a Factorization Machine asks, "Do you really like Sci-Fi Action movies more than just the sum of the two?" It looks for the special "spark" that happens when two features interact.

The researchers asked: What if we taught our physics networks to look for these sparks between variables? Instead of just feeding the network a list of numbers (like $x$ , $y$ , $z$ , and $t$ ), they broke these numbers into separate "fields" and forced the network to calculate how every pair of fields interacts with each other before making a prediction. They built three new versions of these networks:

FM-PINN: For solving specific physics problems on a grid.
FM-Operator: A new type of network that learns the general "rules" of a system, not just one specific solution.
FM-DeepONet: A tweak to an existing popular network (DeepONet) to add this interaction superpower.

What They Found: The Magic of "High-Dimensional" Collisions

When they tested these new networks, the results were a mix of "Wow!" and "Hmm, not so much."

The Big Win: When Things Get Complicated
The new networks shined when the problems got high-dimensional and smooth. Imagine trying to predict the temperature in a 4D or 5D space (which sounds impossible, but in math, it just means tracking many variables at once). In these scenarios, the variables are constantly bumping into each other. The authors found that by explicitly modeling these interactions, FM-PINN reduced the error by a massive 43.98% on a 4D convection-diffusion problem and 55.43% on a 5D linear advection problem compared to the standard version. It was like giving the network a pair of glasses that let it see the hidden connections between the variables.

The Shock Test: Handling the "Jagged" Stuff
Physics isn't always smooth; sometimes it has sharp shocks, like a sonic boom or a sudden traffic jam. The authors tested their networks on equations that create these sharp edges, such as the Burgers equation and the Buckley-Leverett equation (which models oil and water mixing). Here, the FM-Operator and FM-DeepONet were absolute champions.

On a tricky "parametric Burgers" problem (where the fluid's stickiness changes), the new FM-Operator achieved an error of 0.0328, while the standard DeepONet struggled with 0.0794. That's a 58.61% improvement!
Even more impressive, the FM-Operator did this with only 42,826 parameters (the "brain cells" of the network), whereas the standard DeepONet needed 108,161 parameters to try and do the same job. It was like a small, efficient team outperforming a bloated, expensive one.

The "Not So Much" Zone: Smooth and Simple
However, the paper is careful to note that this isn't a magic bullet for everything. When the problems were simple, low-dimensional, or very smooth (like the Helmholtz equation or basic Heat equations in 1D), the new networks didn't always win. In fact, on some of these smooth problems, the standard networks were actually better. The authors suggest that if the variables don't need to "talk" to each other much, forcing them to interact just adds unnecessary noise and confusion. It's like trying to have a deep conversation with someone who just wants to say "hello."

The Verdict: A Specialized Tool, Not a Universal Fix

The paper concludes that these feature interaction modules are a powerful tool, but they are most useful when the physics problem has strong, complex dependencies between its parts. If you are dealing with high-dimensional smooth problems or messy, shock-filled equations, adding this "interaction layer" helps the network understand the physics much better, often with fewer parameters. But if the problem is simple or the variables act independently, the standard approach might still be the way to go.

The authors also compared their work to other advanced methods, like Shift-DeepONet, which tries to move the network's "basis functions" around to follow shocks. They found that their FM-Operator was still more accurate and efficient on the Burgers equation, reducing the error by 28.0% while using 70% fewer parameters.

In short, this paper suggests that teaching neural networks to explicitly model how their inputs "date" each other is a promising direction for solving the hardest physics problems, especially those involving sharp changes or many interacting variables. It's not a one-size-fits-all solution, but for the right kind of messy, complex physics, it's a game-changer.

Technical Summary: Feature Interaction Modeling for Physics-Informed Neural Networks and Neural Operators

Problem Statement

Solving parameterized partial differential equations (PDEs) efficiently and accurately remains a fundamental challenge in science and engineering. Traditional numerical methods rely on predefined meshes and problem-specific schemes. While Physics-Informed Neural Networks (PINNs) and Neural Operators (e.g., DeepONet, Fourier Neural Operators) offer mesh-free alternatives, existing approaches largely focus on domain decomposition, spectral factorization, or input encoding (e.g., Fourier features). A critical gap remains in modeling the interactions among input features themselves. Although tensor decompositions (CP, Tucker) hint at the importance of cross-feature interactions, standard architectures often treat input coordinates, sensor values, and physical parameters as concatenated vectors without explicitly capturing their pairwise couplings. This is particularly limiting for high-dimensional PDEs and problems with sharp gradients or discontinuities where variable couplings are strong.

Methodology

The authors propose embedding Feature Interaction modules derived from Factorization Machines (FMs) into PINNs and Neural Operators. The core mechanism is the bi-interaction pooling layer, which explicitly models pairwise interactions between semantic fields of the input.

Theoretical Motivation

The approach is motivated by the second-order Taylor expansion of multivariate functions. For a target function $g(F)$ dependent on field blocks $F = (F_1, \dots, F_M)$ , the expansion includes cross-field Hessian terms $\sum_{i<j} \delta F_i^\top H_{ij} \delta F_j$ . The proposed bi-interaction block, defined as $z_{BI} = \sum_{i<j} e_i \odot e_j$ (where $e_i$ are field embeddings and $\odot$ is the Hadamard product), explicitly constructs these mixed-block products. This provides an inductive bias that aligns with the local structure of PDE solution manifolds, particularly where cross-coordinate or sensor-query dependencies are significant.

Proposed Architectures

The paper introduces three specific architectures:

FM-PINN:
- Structure: Treats each coordinate component (e.g., $x, t$ or $x_1, \dots, x_D$ ) as a separate semantic field.
- Mechanism: Each field is embedded into a shared latent space, and pairwise interactions are aggregated via bi-interaction pooling before being passed to an MLP decoder.
- Goal: To capture spatio-temporal variable interactions explicitly, improving approximation for smooth, high-order, and high-dimensional PDEs.
FM-Operator:
- Structure: A direct operator architecture (non-branch-trunk) that treats sensor input functions ( $s$ ), query coordinates ( $y$ ), summary statistics ( $q(s)$ ), and physical parameters ( $\eta$ ) as distinct fields.
- Mechanism: It models cross-group interactions between the input function and the query location. The prediction combines an MLP path operating on pooled cross-field products with a linear path retaining raw first-order effects.
- Goal: To handle nonlinear conservation laws and problems with sharp gradients or discontinuities by modeling how sensor values and query coordinates jointly determine the solution.
FM-DeepONet:
- Structure: Injects the Feature Interaction Block into the standard DeepONet framework.
- Mechanism: It constructs a field set from the branch features ( $b$ ), trunk features ( $\phi$ ), sensor statistics, and parameters. The interaction block refines these features, producing correction terms ( $\Delta b, \Delta \phi$ ) that are added back to the original features before the final inner-product projection.
- Goal: To enhance the standard branch-trunk factorization by allowing pairwise interactions among branch features, trunk features, and auxiliary inputs to jointly refine the prediction.

Key Contributions

Integration of Factorization Machines: The work adapts bi-interaction mechanisms from recommendation systems (FMs/NFMs) to continuous PDE inputs for both PINNs and Neural Operators, extending this approach beyond recent symbolic or expression-based methods like PI-MFM.
Explicit Interaction Modeling: Unlike standard concatenation, the proposed architectures explicitly model pairwise relationships between spatial coordinates, time, sensor values, and physical parameters.
Three Novel Architectures: The development of FM-PINN, FM-Operator, and FM-DeepONet, providing a unified strategy for feature interaction across different PDE solving paradigms.
Parameter Efficiency: FM-Operator achieves superior accuracy on challenging benchmarks with significantly fewer parameters (approx. 43K) compared to standard DeepONet (approx. 108K) and Shift-DeepONet (approx. 141K).

Experimental Results

FM-PINN vs. PINN

High-Dimensional Smooth Problems: FM-PINN demonstrates substantial accuracy gains on multidimensional smooth PDEs. For example, it reduced relative $L_2$ errors by 45.69% on 2D convection-diffusion, 63.26% on 4D linear advection, and 55.43% on 5D linear advection compared to vanilla PINN.
Limitations: The method shows inconsistent or negative gains on 1D problems (where no cross-coordinate pairs exist) and specific 2D problems like Darcy and reaction-diffusion. It does not inherently solve the optimization difficulties associated with shock-dominated 1D Burgers equations under residual-based training, though it performs well on smooth 4D Burgers variants.

Neural Operators (FM-Operator & FM-DeepONet)

Shock and Discontinuity Handling: FM-Operator and FM-DeepONet excel on specific nonlinear conservation laws with shocks (e.g., Buckley-Leverett, Cubic Conservation, LWR, Parametric Burgers). FM-Operator reduced errors by 54.37% on parametric Burgers and 38.06% on cubic conservation compared to DeepONet. However, the method offers no consistent advantage on all nonlinear cases; notably, it underperforms standard DeepONet on the Kuramoto–Sivashinsky equation, where DeepONet achieves the lowest error.
Comparison with Shift-DeepONet: On the viscous Burgers equation ( $\nu=0.001$ ), FM-Operator achieved a 28.0% lower mean error than Shift-DeepONet while using roughly 70% fewer parameters.
Smooth Benchmarks: The advantage is less consistent on smooth operator learning tasks (e.g., Helmholtz, Heat, Poisson). DeepONet often outperforms FM variants on low-dimensional smooth problems, suggesting the interaction mechanism is most valuable when strong cross-field dependencies exist.
Ablation Studies:
- Split vs. Non-Split: Splitting query coordinates into separate fields yielded minimal differences, suggesting sensor-query interactions are more critical than query-query interactions.
- Parameter Count: Reducing DeepONet's capacity did not close the accuracy gap with FM-Operator, indicating the gains stem from the interaction structure, not just parameter count.
- Statistics: Adding sensor statistics (mean, std, max) to standard DeepONet did not reproduce the performance gains of FM models, confirming the benefit arises from the interaction mechanism itself.

Significance and Claims

The paper claims that feature interaction modeling provides a simple and parameter-efficient way to improve PDE learning in scenarios where the solution operator contains strong dependencies between the input function and the query location.

Conditional Utility: The authors emphasize that the method is not a universal improvement. It is particularly effective for high-dimensional smooth PDEs (where cross-coordinate coupling is substantial) and specific nonlinear conservation laws with shocks (where sensor values and query locations must interact to track moving fronts). However, it offers no consistent advantage on smooth operator learning benchmarks and can fail on specific nonlinear cases like Kuramoto–Sivashinsky.
Inductive Bias: The work positions the bi-interaction block as a specific inductive bias that aligns with the second-order structure of PDE solution manifolds, offering a targeted alternative to generic deep learning architectures.
Future Directions: The paper notes that the current results are based on specific benchmarks and suggests future work should explore adaptive field grouping to add interaction terms only where beneficial, and further investigate the numerical rank of target solutions to better predict when this architecture will outperform separable or standard operator models.