An Empirical Investigation of Neural ODEs and Symbolic Regression for Dynamical Systems

This paper demonstrates that combining Neural Ordinary Differential Equations (NODEs) for effective data extrapolation with Symbolic Regression (SR) for equation recovery offers a promising hybrid approach to discovering governing physical laws from limited and noisy data.

The Gist

Imagine you are trying to figure out the rules of a game, but you only have a few blurry, shaky video clips of it being played. You want to write down the exact laws of physics that govern the game, but the data is messy, and you don't have enough footage to see everything clearly.

This paper is about a team of scientists who tried to solve this problem using two different "superpowers" of artificial intelligence: Neural ODEs and Symbolic Regression.

Here is a simple breakdown of what they did and what they found, using everyday analogies.

The Two Superpowers

1. Neural ODEs (The "Intuitive Artist"):
   Think of this as an AI that watches a few seconds of a bouncing ball and learns the feeling of how it moves. It's great at predicting where the ball will go next, even if you haven't shown it that specific spot before. However, it's a "black box." It can tell you where the ball will be, but it can't explain why in simple math terms. It's like a chef who can perfectly recreate a dish by taste but can't write down the recipe.

2. Symbolic Regression (The "Detective"):
   This is an AI that looks at data and tries to find the actual mathematical formula (the recipe) behind it. It wants to find the equation $F = ma$ rather than just predicting the motion. The problem is, this detective needs a lot of clear, high-quality evidence to solve the case. If the evidence is too noisy or scarce, it gets confused.

The Experiment: Two Test Cases

The researchers tested these tools on two different systems:

• The Cart-Pole: Imagine a stick balanced on a moving cart. The scientists wanted to see if the AI could predict how the stick would fall if the cart moved in a new way.
• The Bio-Model: A simulation of bacteria adapting to a change in their food supply. They wanted to see if the AI could figure out the biological rules governing how the bacteria grow.

They added "noise" (like static on a radio) to the data to make it realistic and difficult.

Key Findings

1. The Artist Can Paint Outside the Lines (Extrapolation)

The researchers found that the "Intuitive Artist" (Neural ODE) is surprisingly good at guessing what happens in situations it hasn't seen before, but only if the new situation feels similar to the old ones.

• The Analogy: If you teach an AI how a car drives on a sunny day, it can guess how it drives on a cloudy day because the physics are the same. But if you ask it to drive on the moon, it might fail because the "dynamic similarity" is gone.
• The Result: The AI didn't need to see every single possible starting position. It just needed to see enough types of movement to understand the underlying rhythm. Once it understood the rhythm, it could predict the future accurately, even for times much longer than it was trained on.

2. The Detective Needs the Right Clues (Input Variables)

When the "Detective" (Symbolic Regression) tried to find the math equations from the noisy data, it succeeded, but with a catch: It needed the right ingredients.

• The Analogy: Imagine trying to solve a mystery about a cake. If you only give the detective the flour and sugar, they might guess the recipe. But if the recipe also requires a secret spice (a specific variable) and you don't give them that spice, they will write a wrong recipe.
• The Result: When the researchers gave the AI all the necessary variables, it found the correct equations. When they hid a key variable, the AI got confused and wrote a simplified, incorrect version of the law.

3. The Magic Combo: Using the Artist to Help the Detective

This is the most exciting part. The researchers realized that the "Intuitive Artist" (Neural ODE) is so good at smoothing out messy data that it can act as a cleaner for the "Detective."

• The Strategy:
  1. Take a tiny amount of real, noisy data (only 10% of what you usually need).
  2. Train the "Artist" on this small bit.
  3. Let the "Artist" generate a huge, clean, perfect dataset based on what it learned.
  4. Feed this clean dataset to the "Detective."
• The Result: Even though the "Detective" only saw 10% of the original data (via the Artist's generation), it managed to recover two out of three of the correct governing equations and a very good guess for the third.
• Why it worked: The "Artist" acted like a noise-canceling headphone. It filtered out the static and revealed the true signal, making it much easier for the "Detective" to find the math.

The Bottom Line

The paper suggests a new way to do science when you don't have a lot of data:

1. Use a flexible AI (Neural ODE) to learn the "vibe" of the system from a small, noisy sample.
2. Let that AI generate a clean, full picture of the system.
3. Use a formula-finding AI (Symbolic Regression) to read that clean picture and write down the actual laws of physics.

It's like using a skilled sketch artist to fill in the missing details of a blurry crime scene photo, so the detective can finally read the license plate and solve the case. This approach could be a powerful tool for scientists working in fields where data is hard to get.

Technical Summary

Technical Summary: An Empirical Investigation of Neural ODEs and Symbolic Regression for Dynamical Systems

Problem Statement
Accurately modeling the dynamics of complex systems and discovering their governing differential equations are fundamental to scientific discovery. However, a significant challenge exists in leveraging experimental data, which is often noisy and sparse, to infer these dynamics. While Neural Ordinary Differential Equations (NODEs) offer a powerful continuous-time modeling approach, their performance under noisy conditions and their ability to extrapolate to new boundary conditions remain under-explored. Conversely, Symbolic Regression (SR) can discover exact governing equations but typically requires large, high-quality datasets that are difficult to obtain in experimental settings. This work addresses the gap between these two approaches by investigating whether NODEs can serve as a data augmentation tool to enable SR to infer physical laws from limited, noisy data.

Methodology
The study utilizes noisy synthetic data from two distinct damped oscillatory systems:

1. Cart-Pole System: A mechanical system governed by the angle dynamics of a pole on a cart, simulated with added uniform noise (±5%).
2. Bio-model: A biological model describing bacterial adaptation to changing nutrient environments, governed by three coupled ordinary differential equations involving state variables $\psi_A$, $\phi_R$, and $\chi_R$.

The methodology involves a two-stage pipeline:

• NODE Training and Evaluation: NODEs were trained on subsets of the simulation data (ranging from 10% to full datasets) with varying sampling frequencies. The models were evaluated on their ability to interpolate and extrapolate to unseen initial conditions and time horizons. The JAX-based Diffrax library was used for implementation.
• Symbolic Regression: The PySR library was employed to attempt the recovery of the ground-truth equations. SR was tested on two types of datasets:
  1. Direct ground-truth simulation data (with and without noise).
  2. Full datasets generated by a NODE trained on only 10% of the original simulation data.
     The analysis specifically examined the impact of input variable selection (e.g., including the auxiliary variable $\lambda$) and the presence of noise on the recovery of the equations.

Key Results

• NODE Extrapolation Capabilities: NODEs demonstrated effective extrapolation to new boundary conditions, provided the resulting trajectories shared dynamic similarity with the training data.
  • In the Cart-Pole system, low Mean Squared Error (MSE) was observed outside the training region for points lying on the same phase space trajectories as the training data.
  • In the Bio-model, a model trained exclusively on "up-shift" nutrient changes successfully predicted "down-shift" responses with less than 5% error, despite never seeing down-shift data during training.
  • High-quality long-term predictions (up to 8 hours) were achievable even with sparse training data (as few as 10 data points per hour), whereas interpolation error increased significantly with extremely sparse sampling (e.g., 5 points/hour) due to noise sensitivity.
• Symbolic Regression Performance:
  • Ground Truth Data: SR successfully recovered all three governing equations from noise-free data when the auxiliary variable $\lambda$ was included as an input. However, with 5% noise, SR failed to recover the full structure of the most complex equation (Equation 2), finding only a drastic simplification.
  • NODE-Augmented Data: When SR was applied to data generated by a NODE trained on just 10% of the original simulation, it successfully recovered two of the three governing equations (Equations 3 and 4) and provided a good approximation for the third (Equation 2).
• Denoising Effect: The study observed that the NODE acted as a denoising filter. While SR struggled to recover the true structure of Equation 2 from noisy ground-truth data, the NODE-generated data allowed SR to find a better approximation, effectively compensating for the noise by absorbing small signal terms into the discovered constants.

Significance and Claims
The authors claim that this work highlights a promising new approach for scientific discovery, particularly in data-scarce domains. The primary contribution is the demonstration that NODEs can learn underlying dynamics from limited, noisy data and generate enriched datasets that enable Symbolic Regression to infer physical laws.

The paper modestly concludes that while the pipeline successfully recovered two out of three equations and approximated the third, there is room for improvement. The authors suggest that future work could enhance these results by:

• Extending SR analysis to diverse, multi-condition data rather than single-shift simulations.
• Optimizing NODE training data to maximize generalization.
• Incorporating physical priors (e.g., unit matching) or alternative frameworks like SINDy into the SR search.
• Exploring more advanced architectures such as Neural Controlled Differential Equations (Neural CDEs).

Ultimately, the research posits that using NODEs to enrich limited experimental data represents a viable strategy for enabling symbolic regression to discover governing equations where traditional methods might fail due to data scarcity or noise.