🤖 machine learning

Balancing Plasticity and Stability with Fast and Slow Successor Features

This paper demonstrates that in naturalistic, continually changing environments, stabilizing Successor Features through multi-timescale synaptic consolidation outperforms plasticity-focused methods, suggesting that preserving predictive representations is crucial for balancing stability and adaptation in deep reinforcement learning.

Original authors: Raymond Chua, Doina Precup, Blake Richards

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Raymond Chua, Doina Precup, Blake Richards

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to walk. In a perfect world, the floor is always flat, your shoes never change, and your legs stay the same size. But in the real world, things are messy. Sometimes the floor gets slippery, sometimes it's icy, and sometimes your shoes get heavier or lighter. You have to keep walking without falling, even as the ground beneath you slowly shifts.

This paper tackles a big problem in Artificial Intelligence (AI): How do we teach robots (or AI agents) to keep learning when the world around them is constantly, slowly changing?

Here is the breakdown of their discovery, using simple analogies.

The Big Problem: The "Stability vs. Plasticity" Dilemma

Think of an AI agent's brain as a student taking notes.

Plasticity is the ability to write down new notes quickly. If the student is too plastic, they write down everything new but immediately forget what they wrote yesterday.
Stability is the ability to keep old notes safe. If the student is too stable, they remember everything perfectly but can't learn anything new because they refuse to change their notes.

Most AI research has focused on sudden changes (like a student suddenly switching from math class to art class). But the real world is more like a slow drift (like the weather getting gradually colder over a week). The authors found that in these slow-drifting situations, stability is the real hero. Agents that tried to be "super flexible" (resetting their brain parts) actually failed. They needed to hold onto what they knew while slowly adapting.

The Solution: "Successor Features" as a Map

The authors didn't just try to make the AI's brain more stable; they changed what the AI was memorizing.

Imagine you are navigating a city.

Standard AI (Q-values): This is like memorizing specific turn-by-turn directions: "Turn left at the red house, then right at the bakery." If the bakery moves, your directions are useless.
Successor Features (SFs): This is like memorizing the layout of the city and the relationships between places. "The bakery is usually near the park." If the bakery moves, you still know the park is there, and you can figure out where the bakery is now relative to the park.

The paper argues that it's much easier to stabilize a "map of relationships" (Successor Features) than a list of specific directions.

The Secret Sauce: The "Fast and Slow" Memory System

The authors combined these "maps" with a biological concept called Synaptic Consolidation. They built a memory system that works like a chain of buckets with different sizes:

The Fast Bucket (Plasticity): This is a small, open cup. It catches new information immediately. It changes fast, allowing the AI to react to the slippery floor right now.
The Slow Buckets (Stability): These are large, heavy barrels connected to the cup. Water (information) flows slowly from the cup into the barrels. Once the water is in the barrel, it stays there for a long time. This preserves the "map" of the city even when the weather changes.

By having multiple buckets (timescales), the AI can react quickly to a sudden slip (using the cup) while keeping the long-term structure of the world safe in the barrels.

What They Found

They tested this in two ways:

A Slippery Room: An agent had to navigate a room where the floor randomly became icy, causing it to slide in the wrong direction.
A Robot Walker: A robot (like a Humanoid or a Cheetah) had to walk while its own body weight slowly changed, making it heavier or lighter.

The Results:

Stability Wins: Agents that tried to constantly "reset" their brains to be flexible performed poorly. They forgot too much.
Maps Beat Directions: Agents that stabilized the "maps" (Successor Features) performed much better than those that just tried to stabilize the "directions" (standard Q-values).
The Multi-Speed System is Best: The system that used the "Fast and Slow" buckets (Synaptic Consolidation) on the "Maps" (Successor Features) was the clear winner. It learned the fastest and didn't forget the old rules.

The Takeaway

When the world changes slowly and naturally, you don't need a brain that is constantly wiping the slate clean. You need a brain that has fast reaction times for immediate changes but deep, slow memories to keep the big picture stable. By teaching AI to memorize the "structure" of the world rather than just specific outcomes, and by protecting that structure with a multi-speed memory system, we can build agents that adapt to a drifting world without falling apart.

Technical Summary: Balancing Plasticity and Stability with Fast and Slow Successor Features

Problem Statement

Deep Reinforcement Learning (RL) agents often struggle in non-stationary environments. While prior research has addressed the "stability-plasticity dilemma" through abrupt task shifts (e.g., sequential Atari tasks), real-world environments typically evolve through naturalistic, continual non-stationarity. In such settings, dynamics shift gradually rather than abruptly, and there are no explicit task boundaries. Standard RL techniques, including those relying on experience replay or abrupt parameter resets, often fail to adapt effectively or suffer from catastrophic forgetting in these continuous drift scenarios. Furthermore, existing approaches to stability and plasticity primarily operate at the level of Q-values or policies, leaving the role of predictive representations largely understudied in this specific context.

Methodology

1. Evaluation Protocol: Naturalistic Continual Non-Stationarity

The authors introduce a continual RL evaluation protocol characterized by smooth, continuous non-stationarity without explicit task boundaries. This is instantiated in two domains:

Slippery 3D Four Rooms: A navigation task where action outcomes are stochastically replaced based on a noisy sine function (simulating wet/icy conditions).
MuJoCo Control Suite: Continuous control tasks (Humanoid, Walker, Quadruped, Half-Cheetah) where the embodiment's mass varies over time via noisy sine functions, non-periodic sine functions, and Ornstein–Uhlenbeck (OU) processes.

2. Core Mechanism: Successor Features (SFs) with Synaptic Consolidation

The paper proposes a framework that combines Successor Features (SFs) with neuro-inspired Synaptic Consolidation (SC) across multiple timescales.

Successor Features: SFs decompose the state-action value function $Q(s, a, w)$ into a predictive representation $\psi(s, a)$ and reward parameters $w$ . Unlike standard Q-learning, SFs capture the expected discounted occupancy of future features, enabling transfer across tasks with shared dynamics. In this work, both the basis features and the SFs must adapt to the changing transition dynamics $p(s'|s,a; \omega_t)$ .
Synaptic Consolidation (SC): Inspired by Benna & Fusi (2016), SC models synaptic strength stabilization as a chain of $K$ $K$ interacting variables ( $u_1, \dots, u_K$ $u_{1}, \dots, u_{K}$ ) with varying capacities ( $C_k$ $C_{k}$ ) and flow strengths ( $g_{k,k+1}$ $g_{k, k + 1}$ ).
- Fast Timescale ( $u_1$ ): Corresponds to visible synaptic efficacy, allowing rapid adaptation to new experiences.
- Slow Timescales ( $u_2, \dots, u_K$ ): Act as hidden variables that interact bidirectionally with neighbors, preserving stable knowledge and preventing interference.
- Implementation: The authors map these variables to SF parameters ( $\psi_{u_k}$ ). The most plastic variable ( $\psi_{u_1}$ ) is updated via standard gradient descent on the Q-SF-TD loss. The remaining variables are updated analytically using Euler's method to simulate continuous dynamics. Crucially, the authors demonstrate that these updates must be performed using Stochastic Gradient Descent (SGD) rather than adaptive optimizers (like Adam) to preserve the relative timescale information inherent in the consolidation dynamics.

3. Baselines and Comparisons

The study compares the proposed SF + SC method against:

Plasticity-focused methods: Parameter resets (P-last, resetting the last layer) and Continual Backprop (CBP, resetting least-used weights).
Stability-focused methods: Elastic Weight Consolidation (EWC) and Synaptic Consolidation (SC) applied to Q-values (TD3+SC, DQN+SC) rather than SFs.
Trust-Region methods: Proximal Policy Optimization (PPO).

Key Contributions

Naturalistic Continual Non-Stationarity Protocol: The authors introduce a benchmark using smooth, stochastic drift processes (periodic/non-periodic sine, OU) in both navigation and continuous control domains, moving beyond abrupt task-switching paradigms.
Diagnosis of the Bottleneck: Through systematic comparison, the paper provides evidence that performance degradation under continuous non-stationarity is primarily driven by instability rather than insufficient plasticity. Methods preserving stability (SC, EWC) consistently outperform those injecting plasticity (P-last, CBP).
SFs as Superior Consolidation Targets: The study demonstrates that applying synaptic consolidation to Successor Features yields superior performance compared to consolidating Q-values. This suggests that stabilizing predictive representations is more effective than stabilizing value estimates in gradually changing environments.
Multi-Timescale Analysis: The paper introduces a cross-attention mechanism to analyze the relative contributions of fast and slow consolidation variables. Results indicate that while fast timescales drive most learning, slower timescales provide complementary stability, and a combination of multiple timescales (e.g., 6–9 variables) is crucial for optimal performance.

Experimental Results

Stability vs. Plasticity: In both the Slippery Four Rooms and MuJoCo environments, stability-preserving methods (SC and EWC) significantly outperformed plasticity-injection methods (P-last, CBP). Agents lacking stability failed to learn effectively in the continual non-stationary settings.
SF + SC Superiority: The combination of SFs with Synaptic Consolidation (SF + SC) achieved the highest performance and learning efficiency. This held true across periodic, non-periodic, and OU drift processes.
Magnitude of Change: The benefit of consolidating SFs (vs. Q-values) became more pronounced as the magnitude of environmental change increased (from mild 25% to severe 100% mass perturbations). Under mild changes, Q-value consolidation was competitive, but under severe drift, SF + SC was distinctly superior.
Timescale Importance: Increasing the number of consolidation variables (introducing slower timescales) improved learning efficiency. Cross-attention analysis revealed that while fast variables ( $u_2, u_3$ ) received the highest attention, slower variables contributed significantly to stability.
Parameter Efficiency: The SF + SC approach outperformed significantly larger baseline networks (e.g., TD3 with increased capacity), indicating that the gains stem from the consolidation dynamics and predictive representations rather than mere model scaling.
Optimization Constraints: The authors note that the method requires SGD; using adaptive optimizers like Adam disrupts the timescale preservation. Additionally, the analytical updates introduce computational overhead, reducing training throughput (FPS) compared to standard baselines.

Significance and Claims

The paper claims that for deep RL agents operating in naturalistic, continually changing environments, stability is often the primary bottleneck, and maintaining plasticity through parameter resets is insufficient. The proposed solution—multi-timescale synaptic consolidation of predictive representations (SFs)—offers a principled approach to balancing the stability-plasticity trade-off without requiring explicit task boundaries or task statistics.

The authors position this work as a step toward understanding how biological systems (via synaptic consolidation) might solve continual learning problems in dynamic settings, suggesting that stabilizing predictive structures across multiple timescales is an effective strategy for robust adaptation. They acknowledge limitations, including the restriction to SGD-based updates and the computational cost of non-parallelizable analytical updates, but emphasize the method's parameter efficiency and robustness in the face of gradual environmental drift.