Learning the Limits of Discovery: Distinct Failure Signatures of SINDy, Neural ODEs, and PINNs Under Sparse and Noisy Observations
This paper systematically evaluates the distinct failure signatures of SINDy, Neural ODEs, and PINNs under sparse and noisy observations, revealing how noise, sampling geometry, and optimization settings differentially impact their precision, stability, and predictive accuracy across various dynamical systems.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Nature is full of systems that change over time: the swirling of weather patterns, the beating of a heart, the rise and fall of animal populations. Scientists have long sought to write down the simple rules that govern these changes, usually in the form of mathematical equations. In an ideal world, we could measure every detail of a system perfectly and feed that data into a computer to reveal the hidden laws. But the real world is messy. Sensors fail, measurements are skipped, and noise creeps in. When data is incomplete or fuzzy, the tools we use to find these rules often stumble. A recent study set out to understand exactly how and why three popular methods for discovering these laws fail when the data is imperfect, revealing that each tool breaks in its own unique way.
The researchers focused on three distinct approaches used to reverse-engineer the rules of motion. The first, known as SINDy, acts like a detective looking for a sparse set of clues; it sifts through a massive list of possible mathematical terms to find the few that actually describe the system. The second approach, called a Neural ODE, uses a flexible computer network to learn the shape of the motion directly, without trying to write a specific equation. The third, known as a PINN, tries to learn the system by forcing a computer model to obey the laws of physics while also fitting the noisy data points. To test these methods, the team generated perfect data for three famous systems: the Lorenz system, which models chaotic weather-like behavior; the Van der Pol oscillator, which mimics a self-sustaining heartbeat; and the Duffing oscillator, which represents a spring being pushed and pulled. They then deliberately corrupted this data, adding random noise and removing most of the points to simulate sparse, real-world observations.
The results showed that the methods do not fail in the same way. The SINDy approach proved surprisingly good at remembering the correct rules, even when the data was very sparse. However, as the observations became fewer, it began to invent false rules, filling the final equation with extra terms that did not belong. The severity of this mistake depended heavily on how the data was missing. When the missing points were scattered randomly, the method struggled significantly. But when the missing points formed a regular, evenly spaced pattern, the method performed much better, recovering the correct rules with high accuracy. The researchers found that simply adjusting a setting that controls how strict the method is about accepting new terms could fix much of this problem, turning a messy, incorrect equation back into a clean, accurate one.
The Neural ODE method behaved differently. On the non-chaotic systems, its predictions simply got worse as the noise in the data increased, a predictable decline. However, on the chaotic Lorenz system, the method initially appeared to collapse, producing flat, static predictions instead of the expected wild swings. The researchers suspected this was because the system was too chaotic to predict over long periods. Yet, a closer look revealed a different cause: the computer had simply not been given enough time to learn. When the researchers increased the number of calculation steps, the method recovered its ability to predict the chaotic motion, even over longer time windows. The failure was not a fundamental limit of the system, but a sign that the training process needed more computational effort.
The PINN method showed a mix of resilience and fragility. It was very good at finding the correct physical parameters when the data was noisy, provided the system was not too chaotic. However, when faced with sparse data on the chaotic Lorenz system, it became unstable. The results varied wildly depending on how the computer started its calculation; one run might find the correct answer, while another run with the same data would fail completely. This suggests that for chaotic systems with few data points, the method lacks a firm anchor to guide it to the right solution. The researchers also tested these methods on a historical dataset of lynx and hare populations from the Hudson Bay Company. Here, the SINDy method again struggled with the sparse, noisy data, inventing incorrect terms that caused the predicted population to diverge into impossible negative numbers, while a PINN constrained to the known structure of the problem found a stable solution.
Ultimately, the study demonstrates that there is no single best tool for discovering the laws of nature from imperfect data. Each method has a distinct signature of failure. SINDy tends to lose precision and invent extra terms when data is sparse, but this can often be fixed by changing how the data is sampled or how strict the selection is. Neural ODEs may appear to fail on chaotic systems, but this is often just a sign that they need more computing power to train properly. PINNs are robust against noise but can become unpredictable when data is too sparse to guide them through chaotic motion. The key takeaway is that understanding how a method fails is just as important as knowing how well it works, and the specific way data is collected—whether randomly or in a regular pattern—can make the difference between a useful discovery and a broken model.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.