🔭 astrophysics

NeuralDMD: Interpretable Neural Representation of Dynamics from Sparse and Noisy Measurements

NeuralDMD is an interpretable, untrained framework that combines neural implicit representations with Dynamic Mode Decomposition to reconstruct and forecast continuous spatio-temporal dynamics directly from sparse, noisy, and indirect measurements without requiring ground truth data or numerical simulators.

Original authors: Ali SaraerToosi, Renbo Tu, Esther Y. H. Lin, Kamyar Azizzadenesheli, Aviad Levis

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ali SaraerToosi, Renbo Tu, Esther Y. H. Lin, Kamyar Azizzadenesheli, Aviad Levis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to understand a complex dance by only seeing a few blurry snapshots of the dancers, taken from a distance, through a foggy window, and with some of the photos missing entirely. This is the daily reality for scientists trying to image the invisible or the distant. Whether they are tracking a storm system across a continent with only a handful of weather stations, or trying to take a video of a black hole using a telescope the size of the Earth, they face a massive puzzle: how do you reconstruct a smooth, moving picture from scattered, noisy, and incomplete clues?

To solve this, scientists often rely on two main tools. The first is "simulation," where they build a computer model of how the world should work based on known physics. But this is like trying to predict the dance by watching a different dance in a different room; if the real dance has a twist the model didn't expect, the prediction fails. The second tool is "neural networks," which are like super-smart pattern-matching machines that can learn from huge amounts of data. But these machines are notoriously bad at guessing what happens next if they haven't seen that specific move before, and they often produce "hallucinations"—making up details that aren't there—when the data is too sparse. The big question in this field is: Can we build a system that doesn't need a pre-written rulebook (simulation) or a massive library of past dances (training data), but can still figure out the rhythm and predict the next step just by looking at the few clues it has?

Enter NeuralDMD, a new method that acts like a detective who can listen to a few scattered notes of a song and instantly figure out the entire melody, the tempo, and even predict how the song will end. The researchers behind this work, from the University of Toronto and NVIDIA, have created a tool that combines the flexibility of modern AI with the logical structure of classic physics. Instead of trying to memorize the whole dance, NeuralDMD breaks the motion down into a few simple, repeating "modes" or patterns. Think of it like realizing that a complex storm isn't just random chaos, but a combination of a few swirling winds and a few steady breezes. By modeling the world as a mix of these simple, predictable patterns, the system can fill in the missing gaps in the data and forecast the future with surprising accuracy.

The paper demonstrates that this approach works wonders in two very different, high-stakes scenarios. First, they tested it on weather data. Imagine trying to map the wind speed across all of North America using only measurements from about 500 weather stations (a tiny fraction of the total area). While traditional methods struggled to guess the wind patterns between the stations, NeuralDMD successfully reconstructed the full, moving map of the wind, capturing the structure of storms and even predicting how they would evolve hours later. Second, they applied it to the Event Horizon Telescope, which tries to image black holes. Because the telescope is actually a collection of radio dishes scattered across the globe, it only captures tiny, scattered fragments of the black hole's "Fourier transform" (a mathematical way of describing the image). Previous methods often produced blurry or static images, but NeuralDMD managed to reconstruct a clear, moving video of the black hole's swirling gas, even predicting how it would look in the future.

Crucially, the authors show that NeuralDMD does this without needing to be trained on millions of simulated black holes or weather maps. It learns directly from the specific, noisy data it is given. However, the paper is careful to note that this magic has a limit: it works best when the underlying motion is somewhat linear, or predictable. When they tested it on highly chaotic, turbulent fluid flows (where the motion is wildly nonlinear), the system's ability to predict the future started to fade, much like trying to predict the path of a leaf in a hurricane. But for many scientific imaging problems where the rules are unknown and data is scarce, NeuralDMD offers a powerful new way to see the invisible and understand the dynamic universe, one sparse measurement at a time.

Technical Summary: NeuralDMD

Problem Statement

Scientific imaging frequently faces the challenge of recovering continuous spatio-temporal dynamics from measurements that are indirect, noisy, and highly sparse. In many scenarios, such as weather data assimilation or black hole imaging via interferometry, ground-truth data is unavailable, and reliable numerical simulators or governing equations (PDEs) are unknown. Existing approaches struggle in this regime:

Simulation-driven models (e.g., operator learning) are biased by training data distributions and fail under out-of-distribution shifts.
Physics-informed neural networks (PINNs) require known governing equations, which limits their utility in discovery settings.
Standard neural fields (e.g., NeRF, MLPs) lack interpretability and overfit easily when trained on sparse data without explicit dynamical priors.
Classical Dynamic Mode Decomposition (DMD) offers interpretability but assumes direct access to dense, grid-based state observations, making it inapplicable to sparse or indirect measurements.

The authors posit a need for a framework that is simulation-free, supports indirect sensing operators, and provides interpretable forecasting without relying on pre-defined physical models.

Methodology: NeuralDMD

The paper proposes NeuralDMD, an untrained (per-instance) reconstruction framework that combines neural implicit representations with the principles of Dynamic Mode Decomposition (DMD).

Core Representation

NeuralDMD models the spatio-temporal field $I(x, y, t)$ as a low-rank linear superposition of spatial modes evolving with simple spectral time dynamics:
$I(x, y, t) = \sum_{j=1}^{r} w_j(x, y) b_j e^{\Omega_j t}$
where:

$w_j(x, y)$ are spatial modes.
$b_j$ are initial state coefficients.
$\Omega_j = \alpha_j + i\omega_j$ are complex spectral coefficients encoding decay rates ( $\alpha_j$ ) and oscillation frequencies ( $\omega_j$ ).

Neural Parameterization

Unlike classical DMD or optimized DMD (optDMD), which operate on fixed grids, NeuralDMD parameterizes these components using lightweight neural networks (MLPs) with trainable weights $\Theta = [\Theta_w, \Theta_\Omega, \Theta_b]$ :

Spatial Modes ( $\Theta_w$ ): A coordinate-based MLP maps spatial coordinates $(x, y)$ to mode values $w_j(x, y)$ . This allows for grid-free evaluation at arbitrary coordinates.
Spectral Coefficients ( $\Theta_\Omega$ ): A network maps a latent vector to the complex spectrum $\Omega_j$ . Crucially, the real part is constrained to be non-positive ( $\alpha_j \leq 0$ ) to enforce physically meaningful decay and prevent exponential growth.
Initial Coefficients ( $\Theta_b$ ): A network maps a latent vector to the initial coefficients $b_j$ .

Optimization Strategy

NeuralDMD is a per-instance method; it does not train on a dataset of simulations. Instead, it fits the model parameters $\Theta$ directly to the observed measurements by minimizing the residual between the model prediction and the data.

Direct Image-Domain: For sparse station observations (e.g., weather), the loss is the $L_2$ difference between observed and predicted values at coordinates $(x_k, y_k, t_k)$ .
Indirect Fourier-Domain: For interferometric data (e.g., Event Horizon Telescope), the model predicts the image, applies a Fourier transform, and matches the result to observed visibilities $V(u, v, t)$ . The loss function incorporates a $\chi^2$ term weighted by measurement uncertainties and includes a marginalization step over complex station gains to handle atmospheric corruptions.

Key Contributions

Novel Representation: The introduction of NeuralDMD, a model-free representation that unifies neural fields with low-rank linear dynamics.
Simulation-Free Optimization: A method to optimize this representation directly on physical data from diverse imaging scenarios (image and Fourier domains) without requiring simulation training or known PDEs.
Interpretability and Forecasting: The framework yields interpretable spatial modes and continuous-time spectra, enabling stable forecasting beyond the observation window.
Empirical Validation: Comprehensive evaluation demonstrating state-of-the-art performance in weather data assimilation and black hole imaging, outperforming baselines in reconstruction accuracy and extrapolation stability.

Experimental Results

The authors evaluated NeuralDMD on two primary tasks:

1. Weather Data Assimilation (Image Domain)

Setup: Reconstructing wind fields from sparse station observations (approx. 500 stations per frame) using ERA5 data.
Baselines: Compared against 3D-Var (variational data assimilation), Vanilla MLP, and SIREN.
Results: NeuralDMD achieved the lowest RMSE and highest PSNR/SSIM at both sparse (~~500) and dense (~~5000) sampling levels. Crucially, it provided stable short-term forecasts (nowcasting) beyond the observation window, whereas 3D-Var degraded over time and MLPs failed to extrapolate.

2. Black Hole Imaging (Fourier Domain)

Setup: Reconstructing spatio-temporal dynamics of Sagittarius A* from sparse, noisy interferometric visibilities (EHT 2017 campaign).
Baselines: Compared against StarWarps, resolve, optDMD, and a spatio-temporal INR (Vanilla MLP).
Results: NeuralDMD produced lower reconstruction errors and sharper, more temporally consistent images. It successfully extrapolated future frames (forecasting) with coherent motion, while the Vanilla MLP diverged rapidly. StarWarps, which fits frames individually, could not extrapolate.
Robustness: The method remained stable under varying noise levels and sparsity. Ablation studies confirmed that parameterizing spectral coefficients via networks (rather than direct optimization) was critical for performance.

3. Limitations and Nonlinearity

The authors tested the method on Navier-Stokes cylinder flows with increasing Reynolds numbers (Re=200 to Re=5200). While NeuralDMD maintained high fidelity during the fitting window, its forecasting accuracy degraded as nonlinearity increased. At high Re, the method's performance converged to that of a simple persistence baseline, indicating the breakdown of the low-dimensional linear assumption for highly turbulent regimes.

Significance and Claims

The paper claims that NeuralDMD represents a significant step toward imaging algorithms capable of scientific discovery rather than just reconstruction. By avoiding reliance on numerical simulators or pre-encoded physical laws, the method allows data to inform the dynamics directly.

Interpretability: The recovered modes and spectra provide physical insights, such as estimating characteristic timescales (e.g., angular velocity of rotating structures in black hole accretion) directly from observations.
Generalizability: The framework naturally extends to arbitrary differentiable measurement operators, making it applicable to a wide range of inverse problems beyond Fourier imaging.
Modesty: The authors acknowledge that the method relies on a linear dynamics assumption. While many complex systems admit accurate low-dimensional linear approximations over finite windows, performance degrades in highly nonlinear regimes. However, for many scientific imaging problems where governing physics are uncertain, this linear surrogate offers a robust and interpretable alternative to black-box neural networks or equation-constrained models.