ADDA: a Modular Framework for Representing, Simulating and Assimilating Dynamics with End-to-end Differentiability
The paper introduces ADDA, a modular, PyTorch-based software framework that enables end-to-end differentiable data assimilation by providing flexible, parallelizable representations of diverse dynamical systems and observation schemes to overcome existing interoperability and implementation challenges.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast, shifting systems of our planet, from the churning atmosphere to the deep ocean currents, scientists face a constant puzzle: how to know what is happening right now when the data is incomplete and the models are imperfect. Nature does not hand over a perfect, real-time map of every temperature, pressure, and current. Instead, researchers rely on two imperfect sources. First, there are computer simulations, which are built on our best understanding of physical laws but are often too simple to capture every detail of a chaotic world. Second, there are observations from satellites, buoys, and weather stations, which are real but often sparse, noisy, and missing huge gaps. To make accurate predictions about the weather or the climate, scientists must blend these two sources together. This process, known as data assimilation, is the art of finding the single most likely state of a system by weighing what the physics says against what the sensors actually see. It is the engine behind modern weather forecasting, yet it has historically been a difficult, manual task that requires experts to write complex, custom code for every new problem they encounter.
A team of researchers at the Helmholtz-Zentrum Hereon in Germany has introduced a new tool designed to remove these barriers, making it significantly easier to combine simulations with observations. They call their creation ADDA, a modular framework that acts as a universal translator for these scientific problems. Before this work, a scientist wanting to test a new way of merging data with a simulation often had to rewrite the simulation code from scratch, ensure it could calculate its own errors, and then manually build the mathematical machinery needed to adjust the model. This process was slow, prone to mistakes, and made it nearly impossible to compare different methods fairly. The new framework changes this by providing a standard set of building blocks. It allows researchers to plug in a simulation, define what kind of data they have, and choose a method to combine them, all without needing to be a specialist in the underlying mathematics of every specific system.
The core innovation of this work is the use of automatic differentiation, a technology originally developed for training artificial intelligence. In the past, to improve a simulation based on observations, scientists had to manually derive and write down the "adjoint" equations—the mathematical reverse-engineering of how a small change in the starting conditions would ripple through the entire system. This was a tedious and error-prone process that often required years of specialized training. The new framework bypasses this entirely. Because the simulations are built using modern software libraries that can automatically calculate these reverse paths, the researchers can simply define the forward physics, and the system automatically figures out how to adjust the starting point to match the observations. This turns a task that once required a team of mathematicians into something a single researcher can do in a few hours.
The researchers demonstrated the power of this approach by testing it on ten different types of dynamic systems, ranging from simple mathematical models to complex representations of ocean and atmospheric flows. In one test, they used a chaotic system known as the Lorenz-63 model, which mimics the unpredictable nature of weather. They showed that the framework could take noisy, incomplete measurements and reconstruct the true, hidden state of the system with high accuracy, even predicting how it would behave after the observations stopped. In another experiment, they tackled a system with two different time scales, where some variables changed slowly while others shifted rapidly. The framework successfully handled this complexity, inferring the fast-moving parts even when they were never directly observed, simply by watching how they influenced the slow-moving parts.
The tool proved flexible enough to handle irregular data, such as observations that arrive at random times rather than on a fixed schedule, and it could work with systems that have external forces, like tides pushing water in an estuary. The researchers also showed that the framework could bridge different software worlds. They connected a simulation written in one programming language to the framework built in another, proving that scientists do not need to rewrite their entire codebase to use these new methods. In a test involving a neural network emulator—a simplified, fast approximation of a complex fluid flow—the framework successfully used the emulator to perform data assimilation, showing that these tools can work even when the original, slow simulation is too difficult to use directly.
Perhaps most importantly, the work includes a sliding-window approach for long-term simulations. In chaotic systems, trying to adjust the starting conditions for a month-long forecast based on data from the end of that month often fails because the errors grow too large. The new framework solves this by breaking the problem into smaller, overlapping chunks. It solves the data blending for the first week, uses that result as a starting point for the second week, and so on. This allows the system to maintain accuracy over long periods without the calculations becoming unstable. The researchers also demonstrated that the framework can operate on powerful graphics processors, which are common in modern computing, making the process fast enough to handle large-scale problems like global ocean modeling.
This work does not claim to have solved every problem in the field of geoscience. The researchers acknowledge that many of the most critical operational models used by weather services today are still written in older programming languages that do not support these automatic calculation features. However, the paper argues that the path forward is clear: either these systems need to be rewritten in modern, differentiable code, or they need to be replaced by neural network emulators that can be trained to mimic them. The new framework provides the necessary infrastructure to make either of those paths viable. By removing the heavy lifting of manual mathematical derivation, the tool opens the door for a wider range of scientists to experiment with new ways of combining data and physics, potentially leading to more accurate forecasts and a deeper understanding of the complex systems that shape our world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.