Empirical Stability Analysis of Kolmogorov-Arnold Networks in Hard-Constrained Recurrent Physics-Informed Discovery

This paper empirically demonstrates that while Kolmogorov-Arnold Networks (KANs) can compete with MLPs on simple univariate residuals in hard-constrained recurrent physics-informed architectures, they suffer from severe hyperparameter fragility, instability in deeper configurations, and consistent failure on multiplicative terms, ultimately revealing limitations in their additive inductive bias for modeling state coupling in oscillatory systems.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Picture: Teaching AI to "Fill in the Blanks" in Physics

Imagine you are trying to predict how a car moves. You know the basic laws of physics: gravity pulls it down, friction slows it down, and the engine pushes it forward. These are the known rules.

However, real life is messy. There might be a weird wind gust, a bumpy road, or a hidden mechanical quirk you don't understand. In the world of science, we call these the "residuals" (the leftover stuff we can't explain yet).

The researchers in this paper built a special AI system (called HRPINN) that acts like a smart mechanic. It already knows the basic laws of physics (the engine and gravity). Its only job is to look at the car and figure out what the mystery forces are doing.

The New Tool: The "Kan" vs. The Old Tool: The "MLP"

For a long time, scientists used a standard AI tool called an MLP (Multi-Layer Perceptron) to solve these mysteries. Think of an MLP as a Swiss Army Knife. It's a general-purpose tool. It can do almost anything, but it doesn't have a specific shape for any one job. It learns by brute force, trying to fit the data however it can.

Recently, a new tool called KAN (Kolmogorov-Arnold Network) was invented. Think of a KAN as a Lego Set designed specifically for building curves.

• The Theory: KANs are built on a math theorem that says any complex shape can be broken down into simple, single-line curves added together.
• The Hope: The researchers hoped that because KANs are built like Legos (modular and structured), they would be much better at finding the "hidden rules" of physics than the Swiss Army Knife (MLP). They thought KANs would be more efficient and easier to understand.

The Experiment: Two Different Cars

To test if the new Lego tool (KAN) was better than the Swiss Army Knife (MLP), they tested it on two very different "cars" (mathematical models of oscillating systems):

1. The Duffing Oscillator (The Simple Car):

   • The Mystery: This car has a problem that is just a simple curve (like a hill). It's a "one-variable" problem.
   • The Result: The KAN did great! It was like using a Lego set to build a simple ramp. It was fast, accurate, and sometimes even better than the Swiss Army Knife.

2. The Van der Pol Oscillator (The Complex Car):

   • The Mystery: This car has a problem where two things interact. Imagine the wind speed changing based on how fast the car is moving. It's a "multiplicative" problem (A $\times$ B).
   • The Result: The KAN crashed and burned. It got confused. It tried to build a complex interaction using only simple, separate Lego pieces, and the structure fell apart. The old Swiss Army Knife (MLP) handled this complex interaction easily.

What Went Wrong? The "Jenga Tower" Problem

The paper found a major flaw in the KAN design when used for these complex, interacting systems.

• The Analogy: Imagine trying to build a tower of Jenga blocks where every block must be perfectly balanced on the one below it.
  • MLP (Swiss Army Knife): It's like a solid brick wall. If you push it, it holds together because the bricks are glued together (dense connections).
  • KAN (Lego/Jenga): It relies on a chain of separate pieces. To understand how "Wind" and "Speed" interact, the KAN has to build a very tall, complex tower of Lego pieces to simulate multiplication.
  • The Failure: In a "recurrent" setting (where the AI predicts the future step-by-step, like watching a video frame-by-frame), tiny errors happen at every step. With the KAN, these tiny errors pile up like a Jenga tower getting wobbly. By the time the AI tries to figure out the complex interaction, the whole tower collapses. The math becomes unstable.

The Conclusion: Don't Throw Away the Lego, But Don't Rely on It Yet

The researchers concluded that:

1. KANs are great for simple, independent problems. If the mystery is just a simple curve, KANs are efficient and accurate.
2. KANs are currently terrible at complex interactions. When two variables need to multiply or interact (like speed $\times$ wind), the standard KAN design is too fragile. It breaks under the pressure of learning step-by-step.
3. The "Swiss Army Knife" (MLP) is still the king for these complex, real-world physics problems right now. It's more stable and reliable.

The Takeaway:
The KAN is a promising new invention, but it's like a race car that only works on a straight track. It struggles on the curves and turns where variables interact. The researchers hope that future versions of KANs will be reinforced to handle these "turns," but for now, if you need to solve complex physics puzzles, the old-school MLP is still the safer bet.

Technical Summary

Here is a detailed technical summary of the paper "Empirical Stability Analysis of Kolmogorov-Arnold Networks in Hard-Constrained Recurrent Physics-Informed Discovery."

1. Problem Statement

The paper addresses the challenge of symbolic discovery of unknown residual terms within Hard-Constrained Recurrent Physics-Informed Neural Networks (HRPINN).

• Context: HRPINN architectures embed known physical laws structurally into a recurrent integrator, forcing the neural network to learn only the unknown residual dynamics. This ensures physical consistency by construction.
• The Gap: While HRPINNs are effective for accuracy, the specific capability of Kolmogorov-Arnold Networks (KANs) to replace standard Multi-Layer Perceptrons (MLPs) in these recurrent, hard-constrained settings for discovering unknown terms remains unexplored.
• Hypothesis: The authors hypothesized that KANs, grounded in the Kolmogorov-Arnold representation theorem (decomposing multivariate functions into sums of univariate functions), would offer superior parameter efficiency and discovery accuracy for physical laws, particularly because their additive inductive bias naturally isolates independent physical contributions.
• Specific Challenge: The study focuses on the boundary between additive separability (where KANs theoretically excel) and multiplicative coupling (where variable interactions occur), testing if KANs can stably learn these interactions in a recurrent loop.

2. Methodology

The authors conducted a rigorous empirical evaluation using the HRPINN framework with two canonical oscillators representing different residual structures:

1. Duffing Oscillator: Residual is a univariate polynomial ($-0.3x^3$). Represents additive separability.
2. Van der Pol Oscillator: Residual involves multiplicative interaction ($(1-x^2)v$). Represents multiplicative coupling.

Experimental Setup:

• Architecture: The residual branch $R_\theta(x, v)$ was implemented as either a standard ReLU MLP or a B-spline KAN. The known physics and integrator were fixed; only the residual branch was trained.
• Training Paradigms:
  • Single-step Teacher Forcing: Standard training where the network predicts the next step based on ground truth.
  • Backpropagation Through Time (BPTT): Full recurrent training to test stability over longer horizons.
• Evaluation Metrics:
  • Test MSE: Standard error metric.
  • Discovery $R^2$: Measures the grid-based correlation between the network's predicted residual surface and the analytical truth over a dense $100 \times 100$ phase space.
  • Unified Candidate Fitting: Instead of relying on KAN-specific symbolic pruning, both MLPs and KANs were fitted to a unified set of candidate functions to isolate the inductive bias effectiveness from the symbolic extraction mechanism.
• Studies: Three complementary studies were performed with 100 random seeds each:
  1. Configuration Ablation: Varying grid size ($G$) and spline order ($k$).
  2. Parameter-Scale Ablation (Teacher Forcing): Comparing efficiency across different model sizes.
  3. Parameter-Scale Ablation (BPTT): Examining the effect of recurrent unrolling on stability.

3. Key Contributions

• Empirical Baseline for KANs in Recurrent Physics: This is the first study to systematically evaluate vanilla KANs within hard-constrained recurrent architectures, establishing a baseline for their suitability in physics-informed discovery.
• Identification of Hyperparameter Fragility: The paper demonstrates that KANs exhibit severe sensitivity to hyperparameters (grid size, spline order) in recurrent settings, leading to high variance and training failures, unlike the robustness of MLPs.
• Differentiation of Inductive Bias Limitations: The study provides evidence that while KANs excel at univariate polynomial recovery (Duffing), their additive inductive bias ($\phi(x) + \phi(v)$) becomes a bottleneck for multiplicative interactions (Van der Pol) in recurrent loops, requiring deep composition that leads to optimization instability.
• Distinction between Expressivity and Optimization: The authors argue that the failure of KANs in these settings is not due to a lack of theoretical expressivity (as KANs can theoretically represent multiplication), but rather due to optimization instability caused by error accumulation in the recurrent integration loop.

4. Key Results

A. Configuration Stability (Table 1)

• Duffing (Univariate): Certain coarse-grid KAN configurations performed competitively with or better than MLPs ($R^2 \approx 0.83-0.86$).
• Van der Pol (Multiplicative): Most KAN configurations failed severely, with many yielding negative $R^2$ values (indicating diverging solutions). MLPs remained robust across all configurations.

B. Parameter Efficiency & Scaling (Table 2)

• Small Models (Teacher Forcing): Very small KANs ($\approx 120$ params) were competitive with similarly sized MLPs on Duffing but collapsed on Van der Pol ($R^2 \approx 0.46$ vs MLP $0.59$).
• Deep Models: Deeper KANs (880 params) became catastrophically unstable under BPTT, with negative $R^2$ and high variance. MLPs scaled gracefully, improving accuracy with parameter count.
• BPTT Impact: BPTT partially stabilized shallow KANs for Van der Pol (improving $R^2$ to $\approx 0.74$), but MLPs still dominated in consistency and accuracy.

C. Qualitative Analysis (Figure 1)

• Duffing: The KAN successfully captured the cubic manifold structure ($x^3$), achieving high local correlation.
• Van der Pol: The KAN failed to resolve the multiplicative structure, often collapsing to a linear approximation rather than the expected parabolic modulation.

5. Significance and Conclusion

• Limitations of Vanilla KANs: The study concludes that while the additive bias of vanilla KANs is beneficial for separable, univariate physical terms, it is insufficient for stable learning of coupled, multiplicative interactions in recurrent physics-informed settings.
• Optimization vs. Expressivity: The primary bottleneck is identified as optimization stability under recurrent error accumulation, not a fundamental lack of expressivity. The deep composition required to approximate multiplication in a recurrent loop leads to instability.
• Future Directions:
  • The authors suggest that hybrid approaches (e.g., operator chaining where $1-x^2$ is modeled separately before interacting with velocity) or specialized KAN variants (e.g., SKANODEs, DeepOKAN) may overcome these stability issues.
  • Future work should focus on stabilizing the learning of interaction terms and integrating automatic symbolic extraction to fully realize the interpretability potential of KANs in physics-informed modeling.
  • The findings serve as a foundational step for extending these architectures to Partial Differential Equations (PDEs) and chaotic systems (e.g., Lorenz attractor).

In summary, the paper provides a critical "reality check" for the application of KANs in recurrent scientific machine learning, highlighting that while promising for specific separable tasks, their current vanilla formulation is fragile and often outperformed by standard MLPs when dealing with complex, coupled dynamics in hard-constrained environments.