Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

1. Problem Statement

The paper addresses the Stability-Plasticity Dilemma in Continual Reinforcement Learning (CRL). While existing methods often focus on preventing Catastrophic Forgetting (retaining performance on old tasks), they frequently suffer from Loss of Plasticity. This occurs when an agent's ability to learn new tasks diminishes after continuous training, even if it retains competence on previous tasks.

The authors argue that the root cause lies in the prevailing Single-Model Paradigm. Most CRL methods preserve a single, highly specialized policy (a "point solution" in parameter space). Under non-stationary conditions and task interference, this single point is often unstable. Crucially, a policy that is optimal for a source task may not be the best starting point (transfer seed) for a new task. By collapsing the solution space to a single representative, these methods discard a "neighborhood" of behaviorally diverse, competent policies that could have served as better foundations for future adaptation.

2. Methodology: TELAPA

The authors propose TELAPA (Transfer-Enabled Latent-Aligned Policy Archives), a framework that shifts the object of preservation from a single policy to a skill-aligned neighborhood of policies.

Core Components:

1. Per-Task Policy Archives (MAP-Elites):

   • Instead of storing one policy per task, TELAPA trains a base policy using PPO and then performs a "post-training illumination" phase.
   • Using MAP-Elites, it generates and stores an archive of diverse, competent "elite" policies for each task. These elites are behaviorally distinct but all achieve high performance on the source task.
   • This creates a library of policy neighborhoods rather than isolated points.

2. Shared Latent Behavior Space:

   • To make these archives comparable across different tasks and time, TELAPA uses a learned trajectory embedder to map policy rollouts into a shared latent space.
   • Policies that are behaviorally similar are mapped to nearby coordinates in this latent space.

3. Archive-Then-Retrieve Mechanism:

   • When a new task arrives, TELAPA retrieves candidate initializations from the union of all previous task archives.
   • It selects candidates based on latent-space diversity and few-shot transfer performance. The goal is to find a source elite whose "basin of attraction" is most recoverable for the new task, not necessarily the one with the highest source-task score.

4. Latent Space Stability (Drift Mitigation):

   • A critical challenge is representation drift: as the agent learns new tasks, the embedder changes, causing previously stored latent coordinates to become misaligned (scrambling the neighborhood structure).
   • TELAPA employs three mechanisms to maintain a navigable latent geometry:
     • Anchor Sets: A bank of reference behaviors frozen from previous tasks.
     • Replay-Based Alignment: A teacher-student distillation loss where a frozen "teacher" embedder guides a live "student" embedder to keep anchor behaviors stable.
     • Periodic Re-embedding: Archives are periodically re-encoded using the current embedder to ensure consistency.

3. Key Contributions

• Reframing Preservation: The paper argues that the relevant object for preservation in CRL is not a single policy vector, but a recoverable neighborhood of behaviorally related policies.
• Demonstration of Source-Optimal vs. Transfer-Optimal Mismatch: The authors provide empirical evidence that the policy optimal for a source task is often not the best seed for a target task, even within the local competent neighborhood. Preserving only the "best" source policy discards better transfer candidates.
• Latent Geometry Maintenance: They introduce a robust procedure for maintaining a shared latent behavior space under non-stationary drift, treating representation stability as a core component of the continual learning problem.
• Stepping-Stone Lineages: The framework preserves "stepping-stone" policies—intermediate solutions that may not be source-optimal but contain structural features useful for future tasks—creating multi-archive lineages that support long-horizon adaptation.

4. Experimental Results

The method was evaluated on a MiniGrid continual learning curriculum involving a sequence of goal-oriented tasks (navigation, pickup, unlocking, clutter manipulation) and their revisits.

• Performance Metrics: TELAPA outperformed a diverse set of baselines (including EWC, L2-Init, Shrink-and-Perturb, DFF, and standard Fine-tuning) on:
  • Mean Success Rate (SR): Highest overall performance (0.706 vs. ~0.59 for the next best).
  • Time-To-Threshold (TTT): Fastest recovery speed on revisited tasks (3.35M steps vs. 5.49M+ for baselines).
  • Coverage & Retention: Successfully learned and retained a higher fraction of tasks.
• Ablation Studies:
  • TELAPA-Static: Removing the online latent space maintenance (fixing the embedder) significantly degraded performance, proving that drift correction is essential.
  • Single-Model Baselines: Methods that stored only one policy per task (e.g., Scratch-Reuse) performed significantly worse, confirming that diversity within the archive is necessary for transfer.
  • Curriculum Robustness: TELAPA maintained its advantage even under "anti-curriculum" (hardest-to-easiest) and "scrambled" task orders, suggesting the gains are not dependent on a specific task sequence.
• Geometric Analysis:
  • Analysis showed that the "good-enough" source set occupies a broad latent basin (median span of 0.792), meaning competent policies are not redundant.
  • Even within a local competent neighborhood, nearby policies differed significantly in their downstream transfer utility.

5. Significance

This work fundamentally shifts the perspective of Continual RL from parameter retention to skill neighborhood preservation.

• Beyond Forgetting: It addresses the often-overlooked "Loss of Plasticity," showing that retaining a single solution can actually hinder future learning.
• Quality-Diversity Integration: By adapting Quality-Diversity (QD) principles (specifically MAP-Elites) to the sequential, interference-heavy setting of CRL, it provides a mechanism to maintain a "library of skills" rather than a single evolving brain.
• Practical Implication: The results suggest that for lifelong agents, the ability to rapidly re-adapt depends on having access to a diverse set of nearby starting points. If the agent collapses this diversity into a single point, it loses the "navigational aid" needed to find the best path for the next task.

In summary, TELAPA demonstrates that plasticity is preserved by maintaining a structured, diverse archive of competent behaviors and a stable latent space to navigate them, rather than by trying to perfect and preserve a single policy.