⚡ electrical engineering

Parameter-Minimal Neural DE Solvers via Horner Polynomials

This paper introduces a parameter-minimal neural architecture for solving differential equations using Horner-factorized polynomials with exact initial condition enforcement and piecewise continuity, which achieves high accuracy with significantly fewer parameters than standard MLP or sinusoidal baselines.

Original authors: T. Matulić, D. Seršić

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: T. Matulić, D. Seršić

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to predict the weather, the path of a falling apple, or the flow of heat through a metal rod. These problems are described by Differential Equations—complex mathematical rules that govern how things change over time and space.

Traditionally, to solve these, scientists use massive, heavy computers or giant neural networks (AI brains) with millions of "neurons" (parameters). It's like trying to solve a simple riddle by building a supercomputer just to read the answer. It works, but it's wasteful, slow, and hard to fit onto small devices like a smartwatch or a drone.

This paper introduces a clever new way to solve these equations using a "Tiny, Super-Smart AI" that is built differently. Here is the breakdown using simple analogies:

1. The Problem: The "Swiss Army Knife" vs. The "Specialized Tool"

Most current AI models are like Swiss Army Knives. They have thousands of tools (parameters) hoping that some combination will solve the problem. They learn by trial and error, trying to guess the answer while also trying to remember the starting rules (like "the apple starts at the top of the tree"). This often leads to mistakes at the start or requires a lot of computing power.

2. The Solution: The "Horner Polynomial" (The Russian Nesting Doll)

The authors propose a new architecture based on something called Horner's Scheme.

The Analogy: Imagine a set of Russian Nesting Dolls. Instead of having a messy pile of 100 separate blocks, you have one big doll that opens to reveal a slightly smaller one, which opens to reveal an even smaller one, and so on.
How it works: This structure allows the AI to represent a complex curve using a very small number of "knobs" (coefficients). It's mathematically efficient, like folding a long piece of paper into a tiny, neat square.
The Result: The AI needs only 10 to 13 knobs to solve problems that usually require thousands. It's the difference between carrying a backpack full of bricks versus a single, perfectly shaped key.

3. The "Hard-Constraint" Trick (Baking the Rules In)

Usually, when you teach an AI, you have to tell it, "Hey, remember to start at zero!" and then hope it learns that rule by getting punished every time it forgets. This is like teaching a child to tie their shoes by yelling "No!" every time they mess up.

This paper does something smarter: They bake the rules into the structure.

The Analogy: Imagine building a toy car where the wheels are glued to the axle. You don't need to teach the car "don't fall off the wheels"; it's physically impossible for it to happen.
How it works: The authors fix the first few "knobs" of their AI so that the Initial Conditions (the starting point) are satisfied perfectly by design. The AI doesn't waste any brainpower trying to remember the start; it only focuses on learning the rest of the journey. This makes it much more accurate and faster to train.

4. The "Spline" Extension (The Train on Tracks)

What if the problem is too long or too twisty for one single "nesting doll" to handle?

The Analogy: Imagine a train traveling across a country. Instead of trying to build one giant, perfect train track for the whole journey, you build short, straight tracks and connect them with smooth joints.
How it works: The authors split the problem into small segments (like train tracks). They use a tiny, simple AI for each segment. At the boundaries where the segments meet, they enforce a rule that the tracks must line up perfectly (continuity) so the train doesn't jump.
The Benefit: This keeps the total number of "knobs" very low but allows the AI to handle very complex, wiggly curves that a single simple model couldn't catch.

5. Why This Matters (The "Lightweight" Revolution)

The paper tested this on standard math problems and even a heat equation (how heat spreads).

The Comparison: They compared their "Tiny AI" against standard "Big AI" models.
The Result: The Tiny AI was more accurate and used 100 times fewer parameters.
Real World Impact: This means we could run high-precision scientific simulations on tiny devices (like a medical implant, a drone, or a smartphone) without needing a supercomputer. It's like fitting a Ferrari engine into a bicycle frame.

Summary

The authors built a specialized, ultra-efficient AI that solves complex physics problems by:

Using a nested mathematical structure (Horner polynomials) to be super efficient.
Hard-coding the starting rules so the AI never makes a mistake at the beginning.
Chaining small models together (like a train) to handle complex paths without getting bloated.

It proves that you don't need a giant brain to solve big problems; sometimes, you just need a very clever, small one.

1. Problem Statement

Differential equations (DEs) are fundamental to science and engineering, but solving them often requires strategies that balance accuracy with computational efficiency, especially for embedded systems, rapid prototyping, or resource-constrained environments.

Limitations of Current Methods: Traditional Physics-Informed Neural Networks (PINNs) and Implicit Neural Representations (INRs) typically rely on large, generic Multi-Layer Perceptrons (MLPs) or sinusoidal networks (SIRENs). These models often require thousands of parameters to achieve high accuracy.
Key Challenges:
- Hyperparameter Sensitivity: Enforcing initial and boundary conditions usually relies on "soft" penalty terms in the loss function, requiring delicate tuning of weights ( $\lambda$ ).
- Derivative Accuracy: Standard activation functions (like ReLU) lack smoothness, leading to poor approximation of higher-order derivatives, which are critical for DE residuals.
- Parameter Efficiency: There is a need for architectures that achieve high accuracy with a drastically reduced parameter count (parameter-minimalism).

2. Methodology

The authors propose a novel architecture that restricts the hypothesis class to Horner-factorized polynomials, embedding physical constraints directly into the model structure rather than relying on loss-function penalties.

A. Core Architecture: Horner Networks (HN)

Instead of a generic MLP, the solution $x(t)$ is represented as a polynomial evaluated via Horner's scheme:
$P_h(t) = a_0 + t(a_1 + t(a_2 + \dots + t(a_{m-1} + a_m t)\dots))$

Structure: The network consists of a cascade of "Basic Horner Blocks" (BHB), where each block performs a single multiply-add operation with a learnable coefficient $a_i$ .
Differentiability: The resulting function is smooth and fully differentiable, allowing for exact computation of derivatives via automatic differentiation.

B. Hard Embedding of Initial Conditions

A critical innovation is the exact enforcement of initial conditions (ICs) by construction:

For an $n$ -th order ODE with ICs $x^{(i)}(0) = x_i$ , the low-order coefficients of the polynomial are fixed to these values (e.g., $a_0 = x(0)$ , $a_1 = x'(0)$ ).
Benefit: This removes the need for penalty terms in the loss function, eliminates hyperparameter tuning for IC weights, and guarantees the solution satisfies ICs exactly, preventing "drift."

C. Training Objective

The model is trained by minimizing the mean-squared residual of the differential equation at collocation points:
$L_{HN} = \frac{1}{M} \sum_{k=1}^{M} \left( F(t_k, N(t_k), N'(t_k), \dots) - f(t_k) \right)^2$
Since ICs are hard-coded, the loss function contains only the residual term, simplifying the optimization landscape.

D. Spline-like Piecewise Extension

To handle complex behaviors or longer intervals without increasing polynomial degree (which can cause Runge's phenomenon), the authors introduce a piecewise Horner model:

The domain is partitioned into subintervals.
A separate small Horner network is trained for each subinterval.
Continuity Constraints: The loss function includes penalty terms to enforce $C^1$ continuity (continuity of the function and its first derivative) at segment boundaries.
Architecture: A demultiplexer selects the active Horner network based on the input $t$ , and a multiplexer combines the outputs.

E. Extension to PDEs

The approach is generalized to Partial Differential Equations (e.g., the Heat Equation) by treating the solution as a multivariate polynomial where coefficients are themselves 1D Horner networks, allowing for efficient multidimensional representation.

3. Key Contributions

Architecture-First, Parameter-Minimal Solver: A neural architecture that solves ODEs/PDEs using a tightly limited set of learnable coefficients (tens of parameters) rather than massive weight matrices.
Hard Constraint Embedding: Initial conditions are embedded structurally, removing the need for loss-weight tuning and ensuring exact satisfaction of constraints.
High Accuracy with Low Parameters: The model achieves high accuracy for solutions and their derivatives using orders of magnitude fewer parameters than standard baselines.
Interpretability: The learnable parameters correspond directly to polynomial degrees of freedom, offering transparent control over model capacity.
Spline-like Extension: A method to improve approximation accuracy on complex domains by using piecewise Horner models with minimal parameter growth.

4. Experimental Results

The authors evaluated their method on three benchmark ODEs (Linear, Nonlinear, and Second-Order Linear) and a 2D Heat Equation, comparing against:

Wide MLP (Leaky ReLU, ~264k parameters)
Compact MLP (Sigmoid, ~106 parameters)
SIREN (Sinusoidal, ~106 parameters)

Key Findings:

Accuracy: The Horner Network (with only 10–13 learnable parameters) achieved the lowest Root Mean Square Error (RMSE) across all benchmarks.
- Example (Type A ODE): Horner RMSE for solution was $4.6 \times 10^{-6}$ , compared to $2.7 \times 10^{-5}$ for SIREN and $8.5 \times 10^{-5}$ for Compact MLP.
- Derivatives: The Horner network significantly outperformed baselines in approximating first and second derivatives, where MLPs with ReLU failed due to lack of smoothness.
PDE Performance: For the 2D Heat Equation, the multidimensional Horner network achieved minimal error with only 45 learnable parameters.
Spline Extension: The piecewise extension improved accuracy by an order of magnitude over the single-block Horner model while maintaining a low parameter count.
Efficiency: The proposed method demonstrated a superior accuracy-to-parameter trade-off, achieving results comparable to or better than networks with 100x–1000x more parameters.

5. Significance and Impact

Resource Efficiency: This work demonstrates that scientific modeling does not require massive neural networks. By leveraging structural priors (polynomials) and hard constraints, high-fidelity solutions can be obtained with extremely low computational overhead, making it ideal for edge devices and real-time control.
Robustness: By removing the reliance on soft penalty weights for initial conditions, the training process becomes more stable and less sensitive to hyperparameter tuning.
Interpretability: The shift from "black box" weights to identifiable polynomial coefficients provides a more transparent model for scientific discovery.
Future Directions: The paper suggests extending this approach to higher-order PDEs, learning optimal subintervals dynamically, and applying it to complex physical systems like fluid dynamics and electromagnetism.

In summary, the paper presents a paradigm shift from "data-driven, parameter-heavy" neural solvers to "structure-driven, parameter-minimal" solvers, proving that carefully designed mathematical constraints within neural architectures can yield superior efficiency and accuracy for differential equation solving.