🤖 machine learning

From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

This paper presents a novel theoretical framework for deep reinforcement learning in continuous environments by modeling actor-critic algorithms as continuous-time stochastic processes, deriving a stochastic differential equation that characterizes the evolution of state distributions in the infinite-width limit, and validating these findings empirically on a toy control task.

Original authors: Saket Tiwari, Tejas Kotwal, George Konidaris

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saket Tiwari, Tejas Kotwal, George Konidaris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: From Ticking Clocks to Flowing Rivers

Imagine you are teaching a robot to walk. In the old way of doing things (standard Reinforcement Learning), the robot takes a step, checks if it fell, takes another step, and checks again. It's like a clock ticking: Tick, check. Tock, check. The robot learns in discrete jumps.

This paper argues that the real world doesn't tick; it flows. A robot's foot doesn't just "jump" from point A to point B; it glides through the air in a continuous stream. The authors want to understand how a robot learns when we stop pretending time is made of ticks and start treating it like a flowing river.

The Problem: The "Two-Clock" Confusion

The authors identify a tricky problem. When a robot learns, it is actually running on two different clocks at the same time:

The Environment Clock (Fast): This is the river flowing. The robot is moving, falling, and getting rewards right now. This happens very fast.
The Learning Clock (Slow): This is the "aha!" moment. After the robot moves for a while, it pauses to think, "Okay, I fell over; I need to change my brain slightly." This is the "gradient step."

The paper asks: How does the robot's brain change while the river is flowing?

Most theories look at the river (the environment) or the brain (the learning) separately. This paper tries to build a bridge between them, showing exactly how a tiny change in the robot's brain affects its movement in the flowing river, and vice versa.

The Solution: The "Infinite Brain" Analogy

To solve this, the authors use a mathematical trick. Imagine the robot's brain is a neural network. Usually, these networks have a fixed number of neurons (like a brain with 100 neurons).

The authors imagine a brain with infinite neurons.

The Metaphor: Think of a single neuron as a grain of sand. A normal brain is a bucket of sand. An "infinite" brain is an entire beach. When you have a whole beach, the individual grains don't matter as much; the shape of the beach becomes smooth and predictable.
The Result: By assuming the brain is infinitely wide, the messy, chaotic math of learning becomes smooth and clean, like a flowing river. This allows them to write down a single, perfect equation that describes how the robot learns.

The "Exploration" Twist

In learning, a robot needs to try new things to see what works. This is called "exploration."

Old Way: Imagine a robot walking in a straight line, but someone randomly kicks it sideways every few seconds. That's "additive noise." It's clumsy.
This Paper's Way: The authors propose a new way to explore. Imagine the robot's own decision-making process is slightly jittery. When it decides to turn left, it might turn left-ish, left-more, or left-less, all at the same time.
The Analogy: Instead of being kicked by an outside force, the robot's internal compass is slightly wobbly. This "wobbly compass" turns out to be a much more efficient way to explore the world and learn faster. They prove mathematically that this "wobbly" method covers the ground better than the "kicked" method.

The Main Discovery: The "Five-Variable" Secret

The biggest claim of the paper is a "closed system."
Usually, predicting how a complex robot learns is like trying to predict the weather: there are too many variables (wind, humidity, pressure, temperature, etc.).

The authors discovered that for this specific type of learning (using their "infinite brain" and "wobbly compass"), you only need to track five things to know exactly what will happen next:

Where the robot is (State).
What the robot is doing (Action).
How fast the robot is changing its action (Action Derivative).
What the robot thinks the future looks like (Value Estimate).
How fast that "future feeling" is changing (Value Derivative).

The Metaphor: Imagine you are driving a car. Usually, to predict where you'll be in 5 minutes, you need to know the engine, the tires, the road, the driver's mood, the weather, etc. The authors found that for this specific type of driving, you only need to know the speed, the steering angle, and the acceleration of the steering wheel. If you know those five things, the rest of the universe falls into place.

The "Toy" Test

To prove this isn't just math on a napkin, they tested it on a simple simulation called "Linear Quadratic Regulator" (LQR).

The Analogy: Think of this as a video game level where a robot just has to balance a pole or walk in a straight line without falling. It's not a complex game like Mario; it's a physics puzzle.
The Result: The robot learned to balance the pole perfectly. Furthermore, the math equation they derived (the "Five-Variable" secret) matched the robot's actual behavior in the simulation almost exactly.

Summary

This paper is a theoretical map. It doesn't build a new robot or a new app. Instead, it provides a mathematical lens to see how learning happens in continuous time.

Before: We saw learning as a series of jerky, discrete steps (ticks).
Now: We see learning as a smooth, flowing process (flows) driven by two clocks (environment and learning).
The Breakthrough: By imagining an infinitely wide brain, they simplified a chaotic problem into a neat, five-variable equation that accurately predicts how an agent learns in a continuous world.

They call this moving "From Ticks to Flows," turning the jagged, confusing steps of learning into a smooth, understandable river.

Title: From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

Problem Statement
Deep Reinforcement Learning (RL) has achieved significant success in continuous control tasks, yet a rigorous theoretical understanding of how neural network-based agents learn in continuous environments remains limited. While theories for deep supervised learning (e.g., Neural Tangent Kernel, mean-field limits) explain the efficacy of overparameterized networks in stationary settings, these frameworks do not directly apply to RL. In RL, the data distribution changes dynamically with gradient steps due to the agent's interaction with the environment. Existing theoretical analyses of continuous control often lack a unified framework that captures the simultaneous evolution of the environment state and the neural network parameters, particularly regarding the stochastic nature of exploration and the continuous-time dynamics of the system.

Methodology
The authors propose a novel theoretical framework modeling deep RL in continuous environments as a continuous-time stochastic process. The core methodology involves:

Continuous-Time Formulation: The problem is cast within a control-affine Markov Decision Process (MDP) where state transitions are governed by Stochastic Differential Equations (SDEs). The authors introduce an "exploratory dynamics" model where both the environment noise and the policy's stochasticity contribute to transition noise, formalized via a specific SDE (Equation 4).
Two-Time-Scale Analysis: The learning process is characterized by two distinct clocks: the environment time ( $t$ ), governing state transitions, and the gradient time ( $\tau$ ), governing parameter updates. The state of the system is treated as a random variable $s_{t,\tau}$ dependent on both time scales.
Linearized Overparameterized Networks: To make the analysis tractable, the authors employ linearized single-hidden-layer neural networks (both for the actor and critic) in the infinite-width limit. The policy and value functions are approximated using a first-order Taylor expansion around their initialization, a common approach in studying the "lazy regime" of deep learning.
Itô-Taylor Expansion: The authors utilize the Itô-Taylor expansion to express the time-dependent state variable as a polynomial in the network parameters. This allows for the tracking of changes in the state distribution over gradient steps.
Stochastic Differential Equations (SDEs): By combining the linearized network dynamics with the continuous-time environment dynamics, the authors derive equations describing the infinitesimal change in the state distribution at each gradient step under a vanishingly small learning rate ( $\eta = O(1/\sqrt{n})$ ).

Key Contributions

Nonparametric Formulation: The paper provides a nonparametric formulation for studying overparameterized neural actor-critic algorithms in continuous state and action spaces.
Closed System of Dynamics: The primary theoretical result (Theorem 6.1) demonstrates that the gradient-time dynamics of the state, action, and value estimates can be described by a closed system of only five time-dependent variables. This system captures the evolution of the state distribution over a single gradient step.
First Derivation of Gradient-Time State Evolution: The authors derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step. This equation accounts for the interplay between the environment's stochasticity and the learning dynamics.
Exploratory Dynamics Equivalence: The paper establishes a numerical scheme that simulates exploratory dynamics in discrete time while remaining faithful to the underlying continuous-time process, proving the equivalence of the proposed exploratory SDE to a system with combined environment and policy noise.

Results

Theoretical Validation: The derived equations show that the changes in the state, action, value estimate, and their derivatives over a gradient step are Gaussian distributed (up to an error term of $O(1/\sqrt{n})$ ) and depend only on the current values of the five key variables.
Empirical Corroboration: The authors validate their theoretical results using a toy continuous control task (Linear Quadratic Regulator - LQR).
- They demonstrate that their proposed exploratory dynamics effectively cover state-action pairs compared to standard additive Wiener noise.
- They show that an episodic actor-critic algorithm using linearized networks successfully learns near-optimal policies in LQR environments with varying state dimensions ( $d_s \in \{1, 2, 8, 32\}$ ).
- The empirical simulation of the theoretical model closely matches the behavior of the online episodic actor-critic algorithm.

Significance and Claims
The paper claims to provide a principled bridge between stochastic control theory and modern overparameterized RL. By recasting deep RL as a two-clock stochastic system, the authors enable local, infinitesimal characterizations of how state, action, and value estimates evolve. The significance lies in:

Offering a tractable theoretical model that explains the learning dynamics of actor-critic algorithms without relying on the full complexity of non-linear, finite-width networks.
Demonstrating that a simplification of overparameterized neural networks emerges from fundamental principles of probability theory and stochastic processes.
Providing a foundation for future theoretical analysis, such as regret and convergence analysis, in continuous control settings.

The authors acknowledge limitations, noting that their results currently rely on smooth dynamics, single-hidden-layer models, and the asymptotic width limit. They state that extending these results to finite-width networks, non-smooth activations, and higher-dimensional proofs are natural next steps, but the current work establishes the foundational nonparametric framework for such inquiries.