💻 computer science

Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations

This paper proposes continuous-time formulations of AdaGrad, RMSProp, and Adam as first-order integro-differential equations and validates their accuracy through numerical simulations, stability analysis, and convergence studies, offering a new theoretical perspective on adaptive optimization methods.

Original authors: Carlos Heredia Pimienta

Published 2026-10-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Carlos Heredia Pimienta

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern artificial intelligence, the ability to learn from data hinges on a fundamental mathematical challenge: finding the lowest point in a complex, multi-dimensional terrain. Imagine a hiker trying to reach the bottom of a valley shrouded in fog. The hiker can feel the slope beneath their feet and takes a step in the direction that goes down. This basic strategy, known as gradient descent, is the engine behind how computers learn to recognize faces, translate languages, or play games. However, the terrain is rarely a simple bowl; it is often a jagged, uneven surface with many small dips and steep cliffs. A hiker who takes steps of a fixed size might get stuck in a shallow dip or bounce wildly off a steep wall. To navigate this effectively, the hiker needs to adapt their stride based on the history of their journey, remembering where the ground was steep and where it was flat.

For years, computer scientists have developed sophisticated tools to help these digital hikers adjust their steps automatically. Algorithms like AdaGrad, RMSProp, and Adam are the standard guides used in this process. They work by keeping a running tally of past movements, using that history to decide how big the next step should be. If a path has been steep and difficult in the past, the algorithm takes smaller, more careful steps. If the path has been flat, it takes larger, bolder strides. While these methods work incredibly well in practice, they are built on a series of discrete, step-by-step calculations. This makes them difficult to analyze with the smooth, continuous tools of mathematics that scientists use to understand how systems evolve over time. Until now, the gap between the jerky, step-by-step reality of these algorithms and the smooth flow of mathematical theory has remained difficult to bridge.

A researcher at IAMM Research in Barcelona has now proposed a new way to view these algorithms, translating their step-by-step nature into a continuous flow. Instead of thinking of the optimization process as a series of distinct jumps, the author models it as a smooth, unbroken motion governed by a specific type of equation that includes a memory of the past. In this new framework, the algorithm's current movement is not just determined by the slope it sees right now, but by an integral of all the slopes it has encountered since the beginning of its journey. This approach treats the algorithm's "memory" not as a simple list of past numbers, but as a continuous weight that influences the present moment. By doing so, the researcher has created a mathematical model that captures the cumulative and adaptive nature of these tools in a single, flowing description.

The core of this work involves rewriting the rules for AdaGrad, RMSProp, and Adam as continuous equations. In the traditional view, these algorithms update their parameters in discrete steps, accumulating information about the gradient, or the steepness of the terrain, at each stage. The new model replaces these steps with a smooth curve where the influence of the past is encoded in an integral term. For the AdaGrad algorithm, this means the current speed is slowed down by the total sum of all past steepness. For RMSProp, the model incorporates a mechanism where older memories fade away exponentially, allowing the algorithm to focus more on recent terrain. For Adam, the most complex of the three, the model tracks two different types of memory simultaneously: one that remembers the direction of the slope and another that remembers how steep it was. The author demonstrates that these continuous equations are not just abstract ideas but accurate approximations of the actual computer code used in practice.

To test this theory, the researcher ran detailed computer simulations comparing the new continuous models against the original, discrete algorithms. The tests were performed on two different types of mathematical landscapes. The first was a simple, smooth valley with a single lowest point, representing a straightforward problem. The second was a more rugged terrain with multiple valleys and a peak in the middle, representing the complex, non-convex problems found in real-world artificial intelligence. In every scenario, the continuous models tracked the behavior of the discrete algorithms with striking precision. When the learning rate, or the size of the steps, was small, the continuous curves and the discrete steps became nearly indistinguishable. The simulations showed that the continuous models correctly predicted how the algorithms would converge to a solution, how they would oscillate before settling, and how their internal memory variables would evolve over time.

The study also delved into the stability and convergence of these continuous models, proving mathematically that they behave as expected under certain conditions. For the simpler, smooth landscapes, the author showed that the continuous models guarantee a steady approach to the lowest point. For the rugged, complex landscapes, the analysis confirmed that the algorithms will eventually stop moving and settle at a critical point, even if that point is not the absolute lowest valley. The research highlights a key difference between the algorithms: those that remember everything, like AdaGrad, accumulate a permanent memory that can slow them down over time, while those that forget, like RMSProp, allow their memory to decay, keeping them more agile. The mathematical proofs confirm that these behaviors are inherent to the structure of the equations and are not just artifacts of the computer simulations.

This work offers a fresh perspective on how we understand the tools that power modern machine learning. By framing these algorithms as continuous flows with memory, the researcher provides a unified language to describe their behavior. This allows for the application of powerful mathematical techniques that were previously difficult to use on step-by-step methods. The findings suggest that the complex, adaptive behaviors of these algorithms can be understood as natural consequences of their continuous dynamics. While the models are currently based on deterministic settings and simplified functions, they provide a solid foundation for future research. The author suggests that this approach could eventually lead to new strategies for designing optimization methods, potentially borrowing ideas from physics and control theory to create even more efficient ways for machines to learn. The simulations confirm that this continuous view is not just a theoretical curiosity, but a reliable map of the terrain these algorithms traverse.

Technical Summary: Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations

Problem Statement
Adaptive optimization algorithms such as AdaGrad, RMSProp, and Adam are fundamental to modern machine learning, yet their theoretical analysis is often hindered by their inherently discrete nature. While continuous-time approximations using ordinary differential equations (ODEs) have been successfully applied to standard Gradient Descent and momentum-based methods, these approaches typically fail to capture the explicit memory effects and cumulative gradient histories that define adaptive methods. The discrete accumulation of past squared gradients (in AdaGrad) or exponentially weighted moving averages (in RMSProp and Adam) introduces nonlocal dependencies that standard ODEs cannot naturally represent. This paper addresses the challenge of formulating a continuous-time framework that accurately models these memory-dependent dynamics to facilitate rigorous stability and convergence analysis.

Methodology
The author proposes modeling AdaGrad, RMSProp, and Adam as first-order integro-differential equations (IDEs). The core methodology involves:

Continuous Formulation: By assuming a relationship between the discrete time step $k$ $k$ and continuous time $t$ $t$ via the learning rate $\alpha$ $α$ (i.e., $t = \alpha k$ $t = α k$ ), the discrete update rules are approximated using Taylor expansions. The author derives that the accumulation of past gradients corresponds to integral operators with specific kernels.
- AdaGrad: Modeled with a uniform kernel, resulting in a cumulative integral of past squared gradients.
- RMSProp: Modeled with an exponential decay kernel, representing the moving average of squared gradients.
- Adam: Modeled with two distinct exponential kernels for the first and second moments, incorporating bias correction terms as time-dependent gain factors.
Theoretical Analysis: The paper employs Lyapunov stability analysis to study the convergence properties of these IDEs. The analysis is divided into two regimes:
- Convex Setting: Assuming $L$ -smoothness and $m$ -strong convexity, the author derives convergence rates and stability bounds.
- Non-Convex Setting: Assuming the objective function is bounded below with compact sublevel sets, the analysis utilizes the Polyak–Łojasiewicz (PL) and Kurdyka–Łojasiewicz (KL) inequalities to establish convergence to critical points.
Numerical Simulation: To validate the continuous models, the author implements a custom IDESolver within the JAX framework. This solver uses an iterative modified method (combining ODE solving with Gaussian quadrature for the integral terms) to approximate the solutions of the derived IDEs. These results are compared against the original discrete algorithms on both convex and non-convex toy models.

Key Contributions
The paper makes three primary contributions:

Continuous Formulations: The author provides the first continuous-time representations of AdaGrad, RMSProp, and Adam as integro-differential equations (Propositions 1–3). These formulations explicitly encapsulate memory effects within nonlocal integral terms, distinguishing them from previous ODE-based models.
Stability and Convergence Theorems:
- For AdaGrad and RMSProp in the strongly convex case, the paper establishes that the distance to the minimizer decays exponentially once the accumulated memory is shown to be bounded. It distinguishes between the cumulative nature of AdaGrad (where memory converges to a non-zero limit) and the forgetting mechanism of RMSProp (where memory decays to zero), leading to different convergence dynamics for the extended memory functional.
- For Adam, the paper introduces a scalar Lyapunov condition ( $H_{\beta_1, \beta_2} < 2(1-\beta_1)$ ) and a shift-defect estimate to prove conditional stability and convergence to the minimizer.
- In the non-convex case, the author proves asymptotic stationarity ( $\|\nabla f(\theta(t))\| \to 0$ ) for all three algorithms. Under PL or KL assumptions, they derive specific convergence rates (exponential, polynomial, or finite-time) and prove convergence to a single critical point or minimizer.
Numerical Validation: The study provides extensive numerical comparisons (Figures 1–15) demonstrating that the continuous IDE models closely replicate the dynamics of the discrete algorithms. The simulations confirm that as the learning rate $\alpha$ decreases, the agreement between the continuous and discrete models improves, validating the accuracy of the first-order approximation.

Results

Convex Convergence: The analysis reveals that while AdaGrad and RMSProp share a structural form, their memory kernels dictate different long-term behaviors. RMSProp's forgetting term allows for a closed exponential decay of the objective gap and memory variable. In contrast, AdaGrad's cumulative memory leads to a bounded but non-vanishing memory term, though the parameter trajectory still converges exponentially to the minimizer.
Non-Convex Stationarity: The continuous models successfully demonstrate that trajectories remain within compact sublevel sets and converge to the critical set. The paper shows that under the PL inequality, convergence to a minimizer is guaranteed, while under the KL inequality, the convergence rate depends on the exponent $\sigma$ .
Simulation Accuracy: Numerical results show that the continuous IDEs accurately capture the oscillatory behaviors, transient peaks, and stabilization phases of the discrete algorithms across various learning rates and parameter configurations ( $\beta_1, \beta_2$ ).

Significance and Claims
The paper claims that modeling adaptive optimizers as integro-differential equations provides a "novel perspective" that captures the nonlocal dynamics inherent in these methods, which ODEs fail to represent. By bridging discrete algorithms and continuous dynamical systems, this approach allows for the application of advanced mathematical tools (such as Lyapunov analysis and IDE theory) to understand the stability and convergence of adaptive methods.

The author emphasizes that their work offers a unified framework for theoretical understanding, distinguishing itself from prior ODE-based studies by explicitly modeling the influence of past gradients through integral operators. While the current work is limited to deterministic settings and simplified objective functions, the author posits that this framework lays the groundwork for future research into nonlocal Lagrangian and Hamiltonian formalisms in machine learning. The paper concludes that the integro-differential approach is a valid and powerful tool for advancing the theoretical analysis of adaptive optimization, though it acknowledges the need for future extensions to stochastic settings and high-dimensional neural network architectures.

More like this