Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

1. Problem Statement

The paper addresses Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models.

• Context: VLA models are emerging as general-purpose embodied agents capable of zero-shot generalization. However, they are brittle when deployed in evolving environments where they must learn new tasks sequentially without access to past data.
• The Challenge: Conventional wisdom in continual learning suggests that Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting (degradation of performance on previous tasks). To mitigate this, complex CRL strategies (e.g., regularization, experience replay, parameter isolation) are typically required. These methods often introduce a stability-plasticity dilemma, where preserving old knowledge hinders the ability to learn new tasks (loss of plasticity).
• The Gap: It is unclear how these complex CRL strategies interact with large-scale pre-trained VLAs, parameter-efficient fine-tuning (PEFT) methods like LoRA, and on-policy Reinforcement Learning (RL).

2. Methodology

The authors conducted a systematic empirical study to evaluate CRL methods on large pre-trained VLAs.

• Experimental Setup:

  • Models: Three different VLA architectures (including OpenVLA-OFT, Pi-0, and OpenVLA).
  • Benchmarks: Five challenging lifelong RL benchmarks (Libero-Object, Libero-Spatial, Libero-Long-Horizon, RoboCasa, and ManiSkill).
  • Training Protocol: All methods used Group Relative Policy Optimization (GRPO) for on-policy RL updates and LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning.
  • Algorithms Compared:
    • Baselines: Sequential Fine-Tuning (Seq. FT) and Multi-Task Training (Oracle).
    • CRL Methods: Elastic Weight Consolidation (EWC), Expert Replay (ER), Dark Experience Replay (DER), Dynamic Weight Expansion (DWE), SLCA (Layer-wise learning rate), and RETAIN (Weight merging).
  • Metrics: Average Success (AVG), Negative Backward Transfer (NBT/Forgetting), Forward Transfer (FWT), and a new metric, Zero-Shot Success (ZS), to measure retention of pre-trained generalization.

• Ablation Studies: To understand why Seq. FT works, the authors removed key components:

  1. Replaced RL with Supervised Fine-Tuning (SFT).
  2. Replaced the large VLA with a small 12M parameter network.
  3. Removed LoRA (performed full fine-tuning).

3. Key Contributions & Findings

A. The "Simple Recipe" Outperforms Complex Methods

Contrary to established beliefs, Sequential Fine-Tuning with LoRA consistently outperformed sophisticated CRL methods across all benchmarks.

• Performance: Seq. FT achieved high average success rates, often matching or exceeding the Multi-Task Oracle.
• Forgetting: It exhibited little to no catastrophic forgetting (NBT < 2%, often negative), whereas complex methods often suffered from reduced plasticity due to their constraints.
• Generalization: Seq. FT preserved and frequently enhanced zero-shot generalization capabilities, outperforming the Multi-Task Oracle on unseen tasks.

B. Mechanistic Analysis: The Synergy of Three Components

The paper identifies that the robustness of Seq. FT arises from a unique synergy between three specific components. Removing any single one causes catastrophic forgetting:

1. On-Policy RL (Objective): Unlike Supervised Fine-Tuning (SFT), which forces the model to match dataset actions regardless of the base policy (causing large KL divergence and forgetting), on-policy RL updates are weighted by the current policy's state distribution. This creates an implicit regularization that prevents the policy from making drastic, uncontrolled shifts away from the pre-trained distribution.
2. Large Pre-trained Models (Capacity): Due to the "curse of dimensionality" in high-dimensional parameter spaces, large models possess a vast "Null Space." Gradient updates for new tasks are likely orthogonal to the directions encoding pre-trained knowledge, minimizing interference. This was validated via Fisher Information Matrix analysis, showing significantly lower interference in large models compared to small ones.
3. LoRA (Constraints): LoRA restricts updates to a low-rank subspace. This prevents specific layers from undergoing disproportionately large structural changes (high-rank updates) that would overwrite pre-trained representations. It ensures adaptation is localized and controlled.

C. Robustness

The effectiveness of Seq. FT was validated under various perturbations:

• Environmental: Camera angles, lighting, and robot starting positions.
• Architectural: Different VLA models (Flow-based vs. Autoregressive).
• Sequential: Different task ordering.
  In all cases, Seq. FT maintained minimal forgetting and high plasticity.

4. Results Summary

• Table 1 & 2: Seq. FT achieved the highest or near-highest Average Success (AVG) across all domains.
• Forgetting (NBT): Seq. FT showed negligible forgetting (e.g., 0.3% on Libero-Spatial), while methods like EWC and RETAIN showed higher forgetting or significantly lower plasticity.
• Zero-Shot (ZS): Seq. FT often improved ZS scores (+5.6% on Libero-Spatial), whereas other methods degraded them.
• Ablation (Table 3):
  • Removing RL (using SFT) caused massive forgetting (NBT ~78%).
  • Using a small model caused massive forgetting (NBT ~11%).
  • Removing LoRA (full fine-tuning) caused massive forgetting (NBT ~40%).
  • Conclusion: All three components are necessary for the "no-forgetting" phenomenon.

5. Significance and Implications

• Paradigm Shift: The paper challenges the necessity of complex CRL algorithms (replay buffers, regularization) for large pre-trained models. It suggests that for large VLAs, simple Sequential Fine-Tuning with LoRA and On-Policy RL is a sufficient and superior strategy.
• Stability-Plasticity Resolution: The findings suggest that the stability-plasticity dilemma is not inherent to continual learning but is a result of using small models or inappropriate training objectives (like SFT). Large models + RL + PEFT naturally resolve this trade-off.
• Scalability: This approach offers a computationally efficient and scalable path for lifelong learning in embodied agents, avoiding the storage overhead of replay buffers and the complexity of parameter isolation.
• Future Directions: The authors suggest future work should focus on improving sample efficiency (bridging the small gap between Seq. FT and the Oracle) and applying these findings to real-world robotic systems via sim-to-real transfer.

In summary, the paper demonstrates that large pre-trained VLAs are "natural continual learners" when combined with on-policy RL and LoRA, rendering complex continual learning mechanisms unnecessary and often detrimental.