Intentional Updates for Streaming Reinforcement Learning

1. Problem Statement

The paper addresses the instability inherent in streaming reinforcement learning (RL), where an agent learns from a continuous sequence of single transitions (batch size = 1) without replay buffers or mini-batches.

• The Core Issue: In standard gradient-based learning, the step size ($\alpha$) is defined in parameter units. However, the relationship between parameters and function outputs (value predictions or policy probabilities) is complex and non-linear. Consequently, a fixed step size does not guarantee a predictable change in the function's output.
• The Consequence: In streaming settings, stochasticity is not averaged out. This leads to "overshooting" (updates that are too large) or "undershooting" (updates that are too small), causing training instability and failure. This phenomenon is referred to as the "stream barrier."
• Limitations of Current Solutions: Existing methods often rely on replay buffers, target networks, or large batch sizes to stabilize learning, which increases computational cost and memory usage. While some streaming methods exist (e.g., StreamX), they often require extensive hyperparameter tuning or auxiliary stabilization techniques.

2. Methodology: Intentional Updates

The authors propose a paradigm shift: instead of specifying a step size in parameter units, they specify an intended outcome in function output units and solve for the step size required to achieve it.

Core Principle

Given a current state $s_t$ and a target quantity $y_t(w)$ (e.g., value prediction $V(s_t)$ or log-probability $\log \pi(a_t|s_t)$), the method defines an intended change $\Delta_t$. Using a first-order Taylor approximation, the step size $\alpha_t$ is derived to satisfy:
$$y_t(w_{t+1}) \approx y_t(w_t) + \Delta_t$$
$$\alpha_t = \frac{\Delta_t}{\nabla y_t(w_t)^\top d_t}$$
where $d_t$ is the update direction (gradient or eligibility trace). This approach is analogous to the Normalized Least Mean Squares (NLMS) algorithm in adaptive filtering but extended to non-stationary RL settings.

Specific Algorithms

The paper instantiates this principle for three key RL tasks:

1. Intentional TD (Value Prediction):

   • Goal: Achieve a fixed fractional reduction of the Temporal Difference (TD) error.
   • Mechanism: The update aims to reduce the current prediction error $\delta_t$ by a factor $\eta$ (where $\eta \in (0, 1]$).
   • Step Size: $\alpha_t = \frac{\eta}{\|\nabla V(s_t)\|^2}$.
   • Extension to TD($\lambda$): For eligibility traces, the method defines the intended outcome as a bounded root-mean-square change over a fading history of predictions, ensuring the error propagates correctly over time without shrinking updates as the trace grows.

2. Intentional Q-Learning (Control):

   • Goal: Apply the same fractional reduction principle to action-value predictions $Q(s_t, a_t)$.
   • Mechanism: Targets a fixed reduction in the Q-learning error.

3. Intentional Policy Gradient (Policy Optimization):

   • Goal: Control the magnitude of the policy change directly, rather than the parameter change.
   • Metric: Uses the change in the log-probability of the sampled action ($\Delta \log \pi$) as a proxy for the local Kullback-Leibler (KL) divergence.
   • Mechanism: The step size is chosen so that the change in log-probability is proportional to the advantage $A_t$, scaled by a meta-parameter $\eta$.
   • Formula: $\alpha_t = \frac{\eta}{\bar{A}_t \|\nabla \log \pi\|^2}$, where $\bar{A}_t$ is a running average of the advantage magnitude.

Practical Enhancements

To make these methods robust for deep RL, the authors integrate:

• Diagonal Scaling (RMSProp-style): Entry-wise normalization of gradients to handle varying scales across network parameters.
• Adaptive Clipping: A dynamic clipping mechanism for TD errors based on their long-term root-mean-square, rather than a fixed range, to handle outliers without biasing the scale.
• Eligibility Traces: Integrated with the intentional step-size calculation to ensure credit assignment over time.

3. Key Contributions

1. Intentional Outcome Principle: A novel framework for step-size selection that targets functional changes (output units) rather than parameter changes, providing a more intuitive and stable learning signal.
2. Algorithmic Family: The development of Intentional TD, Intentional Q-learning, and Intentional Policy Gradient, which are the first deep RL algorithms to successfully operate in a fully streaming regime (batch size 1, no replay) with state-of-the-art performance.
3. Robustness and Transferability: The methods require significantly fewer hyperparameters. A single setting for the meta-parameter $\eta$ (controlling the magnitude of functional change) transfers effectively across diverse environments (MuJoCo, DM Control, Atari, MinAtar) without task-specific tuning.
4. Computational Efficiency: The algorithms are designed for low computational overhead, running efficiently on a single CPU, in contrast to batch-based methods like SAC that require GPUs.

4. Experimental Results

The authors evaluated their methods against strong baselines (StreamAC, StreamQ, PPO, SAC, DQN) across continuous and discrete control domains.

• Continuous Control (MuJoCo & DM Control):

  • Intentional AC (Actor-Critic) achieved stable learning and performance comparable to or better than batch-based PPO and SAC, despite using no replay buffer.
  • It significantly outperformed previous streaming baselines (StreamAC), closing the performance gap between streaming and batch learning.
  • The method was less sensitive to auxiliary stabilization techniques (e.g., reward scaling, input normalization) compared to prior streaming methods.

• Discrete Control (Atari & MinAtar):

  • Intentional Q-learning achieved reliable learning across various games, matching the performance of batch-based DQN and StreamQ.

• Value Prediction:

  • Under a fixed policy, Intentional TD demonstrated lower prediction error and greater robustness to gradient scale variations compared to StreamTD.

• Ablation Studies:

  • Removing components like diagonal scaling or $\delta$-clipping degraded performance slightly, but the intentional scaling mechanism itself was identified as the primary driver of success.
  • The method showed high "update fidelity," meaning the actual change in the function output closely matched the intended change derived from the linear approximation.

• Computational Cost:

  • One update of Intentional AC requires approximately 140x fewer FLOPs than one update of SAC (due to batch size 1 vs. 256 and fewer networks).
  • Even when matched for total compute budget, Intentional AC significantly outperformed SAC.

5. Significance and Impact

• Breaking the Stream Barrier: This work demonstrates that deep reinforcement learning can be stable and effective in a purely streaming setting without the need for replay buffers or target networks. This is crucial for real-time, resource-constrained applications (e.g., robotics, edge devices) where storing experience is impossible or too expensive.
• Interpretability: By defining learning rates in terms of "how much the prediction should change" rather than "how much the weights should move," the meta-parameter $\eta$ becomes more interpretable and transferable across tasks.
• Efficiency: The drastic reduction in computational requirements makes high-performance RL accessible on hardware where GPU-based batch methods are infeasible.
• Theoretical Bridge: The work bridges classical adaptive filtering (NLMS) with modern deep RL, offering a principled solution to the instability of stochastic gradient descent in non-stationary, single-sample environments.

In summary, Intentional Updates provide a robust, efficient, and theoretically grounded approach to streaming deep RL, enabling agents to learn continuously from a single stream of data with performance rivaling complex batch-based systems.