🤖 machine learning

Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation

This paper introduces Polynomial-Augmented Neural Networks (PANNs), a novel architecture that integrates deep neural networks with polynomial expansions and employs orthogonality constraints, basis pruning, and preconditioning to achieve superior accuracy and stability in approximating both smooth and non-smooth functions and solving partial differential equations compared to existing methods.

Original authors: Madison Cooley, Shandian Zhe, Robert M. Kirby, Varun Shankar

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Madison Cooley, Shandian Zhe, Robert M. Kirby, Varun Shankar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern science, researchers often face a dilemma when trying to model the physical world. They need tools that can predict how fluids swirl, how heat spreads, or how structures bend under stress. For decades, two distinct families of mathematical tools have dominated this field. On one side stand the deep neural networks, complex computer programs inspired by the human brain. These are incredibly flexible and powerful, capable of learning patterns in massive amounts of data and handling problems with many variables at once. On the other side are polynomial methods, which use simple, smooth curves to approximate functions. These are precise and efficient when the target is smooth and well-behaved, but they often struggle when the data is messy or the problem becomes too complex. The challenge has always been to find a way to combine the adaptability of the neural networks with the precision of the polynomials without letting one cancel out the benefits of the other.

A team of researchers has now proposed a new architecture that successfully merges these two approaches. They call it a Polynomial-Augmented Neural Network, or PANN. Instead of choosing between a neural network or a polynomial curve, their system uses both simultaneously. Imagine a neural network as a highly skilled artist who can sketch almost any shape, but sometimes misses the fine, smooth details. The researchers added a layer of polynomial functions, which act like a set of perfectly smooth, pre-made curves, to the artist's toolkit. The key innovation is not just adding these curves, but teaching the two parts to work together without stepping on each other's toes. They introduced a set of rules, or constraints, that force the neural network and the polynomial layer to remain distinct, ensuring that the polynomial handles the smooth, global patterns while the neural network focuses on the complex, irregular details. This separation prevents the two components from fighting over the same information, leading to a more stable and accurate model.

The researchers tested this new architecture on a variety of tasks, starting with the ability to perfectly recreate known mathematical shapes. When the target was a smooth polynomial function, the PANN was able to recover the solution with extreme precision, often reaching the limits of what a computer can calculate. This was a significant finding because it proved that adding a neural network did not ruin the polynomial's ability to be exact. In fact, the system could isolate the polynomial part and reproduce it perfectly, even while the neural network was also learning. The team also tested the system on functions that were not perfectly smooth, containing sharp corners or rapid oscillations. Here, the PANN again outperformed standard neural networks and pure polynomial methods. The neural network component successfully captured the jagged, difficult features, while the polynomial layer handled the broader, smoother trends. This complementary behavior allowed the system to approximate difficult functions with fewer errors than either method could achieve alone.

The study also explored how this architecture performs when solving partial differential equations, which are the mathematical equations used to describe physical phenomena like heat flow and fluid dynamics. In these tests, the PANN was used as a physics-informed neural network, meaning it was trained not just on data points but also on the physical laws governing the system. The results showed that the PANN could solve these equations with errors that were orders of magnitude lower than those produced by standard neural networks. For example, in a test involving a two-dimensional Poisson equation, a specific configuration of the PANN achieved an error rate so low it was nearly indistinguishable from the exact solution. The researchers found that the system learned faster and more reliably than traditional methods, reducing the time and computational power needed to reach a high level of accuracy.

However, the researchers were careful to note that the success of this method depends on how the different parts are tuned. They discovered that the choice of rules used to keep the neural network and polynomial layer separate mattered greatly. Some rules worked better for certain types of problems, while others were more efficient for different tasks. They also found that for problems involving sudden jumps or discontinuities, such as a material with a sharp change in properties, the strict rules needed to be relaxed slightly. If the rules were too rigid, the system became too inflexible to handle the sharp changes. This suggests that while the method is robust, it requires careful adjustment depending on the specific problem at hand. The team also addressed the issue of high-dimensional problems, where the number of variables is very large. By using a technique to prune, or remove, unnecessary polynomial terms during training, they prevented the system from becoming overwhelmed by complexity, allowing it to scale to problems with many variables without a massive increase in cost.

The implications of this work extend beyond just better numbers on a screen. The researchers demonstrated that this hybrid approach offers a more reliable way to model the physical world, particularly for tasks where data is noisy or the underlying physics are complex. By combining the strengths of two different mathematical traditions, they created a tool that is both flexible and precise. The experiments confirmed that the neural network does not need to be made larger or deeper to achieve better results; instead, the addition of the polynomial layer provides a structured boost in performance. This suggests a new direction for scientific computing, where the future of machine learning might not be about building bigger neural networks, but about intelligently combining them with classical mathematical tools to create systems that are smarter, faster, and more accurate. The work stands as a demonstration that sometimes the best solution is not to choose between two powerful methods, but to find a way to make them work together in harmony.

Technical Summary: Polynomial-Augmented Neural Networks (PANNs)

Problem Statement

Deep Neural Networks (DNNs) have become a dominant tool in scientific machine learning, particularly for solving partial differential equations (PDEs) via Physics-Informed Neural Networks (PINNs). They offer flexibility, scalability, and the ability to break the curse of dimensionality. However, DNNs face significant challenges, including spectral bias (difficulty capturing high-frequency components), vanishing or exploding gradients, and susceptibility to overfitting. Conversely, traditional polynomial approximation methods offer rapid convergence for smooth functions but suffer from the curse of dimensionality (exponential growth of basis functions) and often require structured grids or oversampling for stability.

The core problem addressed is the lack of a unified architecture that leverages the high-dimensional flexibility of DNNs while retaining the rapid convergence rates of polynomial methods for smooth functions, without reintroducing the computational intractability of high-dimensional polynomials.

Methodology

The authors propose Polynomial-Augmented Neural Networks (PANNs), a hybrid architecture that combines a standard DNN with a trainable polynomial expansion layer. The model output $u_\theta(x)$ is defined as the sum of the DNN output $N(x)$ and a polynomial layer output $P(x)$ :
$u_\theta(x) = N(x) + P(x) = \sum_{j=1}^w a_j \psi_j(x; \theta_h) + \sum_{k=1}^m b_k \phi_k(x)$
where $\psi_j$ are DNN basis functions (activations of the last hidden layer) and $\phi_k$ are polynomial basis functions (specifically Legendre polynomials in this work).

To ensure stable training and enhanced accuracy, the paper introduces three key methodological components:

1. Weak Orthogonality Constraints

To prevent the DNN and polynomial components from learning redundant features, the authors introduce a family of eight discrete orthogonality constraints. These constraints enforce "weak" orthogonality between the DNN basis and the polynomial layer. Instead of enforcing a continuous integral constraint (which is difficult to compute), they enforce discrete constraints on the product of basis functions and coefficients.

Constraint Families: The constraints range from "weak" (e.g., enforcing the product of the total DNN output and total polynomial output is zero) to "strong" (e.g., enforcing orthogonality between every individual DNN basis and every polynomial basis).
Implementation: These are enforced via an additional weighted loss term ( $\lambda_c \|C\|_F$ ) during gradient descent, where $C$ represents the specific constraint matrix (e.g., $N(x)P(x)$ or $a_j \psi_j(x) b_k \phi_k(x)$ ).
Selection: Experiments suggest that constraint CE (enforcing orthogonality of the polynomial layer against DNN basis functions and coefficients) often provides the best trade-off between accuracy and training time.

2. Basis Pruning

To combat the curse of dimensionality introduced by the polynomial component, the authors employ a basis pruning strategy. During training, coefficients ( $a_j$ for DNN, $b_k$ for polynomials) that fall below a specified threshold are zeroed out, and their corresponding basis functions are discarded. This dynamically controls the number of active basis functions, ensuring parsimony and computational efficiency in high-dimensional settings.

3. Preconditioning and Custom Differentiation

Preconditioning: The authors adapt a polynomial preconditioning strategy (originally for least-squares) to both the DNN and polynomial components. A diagonal preconditioning matrix $K$ rescales the basis functions to improve the conditioning of the optimization problem, aiding in the recovery of sparse solutions.
Efficient Differentiation: For PDE applications, the authors utilize a custom forward/backward pass. While DNN derivatives are computed via automatic differentiation, polynomial derivatives are precomputed analytically and stored. This avoids redundant differentiation overhead during training iterations.

Key Contributions

Novel Architecture: The introduction of PANNs, which augment standard DNNs with a single, trainable polynomial layer, viewed either as an adaptive basis method or a residual block with transformed skip connections.
Orthogonality Framework: The formulation of a family of discrete orthogonality constraints that separate the approximation spaces of the DNN and polynomial components, aiding in optimization and reducing redundancy.
Dimensionality Control: The integration of basis pruning and preconditioning to make polynomial augmentation feasible in high-dimensional spaces, addressing the exponential growth of polynomial bases.
Empirical Validation: Comprehensive testing across regression tasks (smooth, non-smooth, noisy, and high-dimensional) and PDE solving (linear, nonlinear, time-dependent, and chaotic).

Results

The paper presents extensive numerical experiments comparing PANNs against standard DNNs, pure polynomial methods (L2 projection and optimized polynomials), and various PINN baselines (including RFF, ModMLP, GradNorm, and Causal).

Polynomial Reproduction: PANNs can exactly recover known polynomial solutions (up to machine precision) when the polynomial degree is sufficient, demonstrating that the DNN component does not hinder the polynomial layer's ability to represent polynomials.
Non-Smooth Functions: PANNs remain competitive with, and often improve upon, either DNNs or polynomial approximations when approximating continuous non-smooth functions (e.g., functions with vanishing derivatives or localized oscillations). The DNN component effectively captures the non-smooth features while the polynomial layer handles the global structure.
High-Dimensional Regression: In synthetic problems up to 6 dimensions and a real-world 8-dimensional housing dataset, PANNs (particularly with ReLU activation and constraint CE) achieved lower relative $\ell_2$ errors than standard DNNs and pure polynomial methods.
PDE Solutions:
- Poisson Equation: PANNs achieved near-machine precision errors ( $\sim 10^{-6}$ ), significantly outperforming standard PINNs of comparable or even larger size.
- Nonlinear/Time-Dependent PDEs: For the Allen-Cahn, Korteweg-De Vries (KdV), Kuramoto-Sivashinsky (KS), and Grey-Scott equations, PANNs consistently reduced relative errors compared to baseline PINNs, though the magnitude of improvement varied by problem and baseline (e.g., achieving a lowest relative error of 0.57 for the chaotic KS equation).
- Limitations: In the Darcy flow problem with jump coefficients (discontinuous solutions), a strongly enforced orthogonality constraint initially degraded performance due to over-regularization. Reducing the constraint strength improved results, suggesting the need for adaptive tuning for non-smooth PDE solutions.

Significance and Claims

The paper claims that PANNs offer superior approximation properties to DNNs for both regression and PDE solution, particularly for smooth functions where polynomials excel. Furthermore, PANNs offer enhanced accuracy or competitive performance over pure polynomial methods when dealing with functions of limited smoothness or high-dimensional data, where polynomials alone struggle.

The authors emphasize that the combination of DNNs and polynomials, governed by weak orthogonality constraints, creates a more favorable optimization landscape (smoother loss surfaces) and allows the model to decompose problems into global (polynomial) and local/high-frequency (neural) components. While the method increases training time due to preconditioning and orthogonality penalties, the resulting accuracy gains and the ability to solve complex PDEs with fewer parameters or iterations justify the cost. The work positions PANNs as a robust, global approximator that inherits the benefits of DNNs (noise robustness, mesh-free nature) while mitigating their spectral bias and convergence limitations.