Technical Summary: CRL-VLA

Problem Statement

The paper addresses the challenge of Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models in open-world robotic environments. While Reinforcement Learning (RL) post-training has proven effective for aligning VLA models with complex manipulation tasks, sustaining this adaptability across non-stationary task streams requires balancing stability (retaining previously learned skills) and plasticity (acquiring new ones).

Existing CRL methods face significant limitations when applied to large-scale, multi-modal VLA architectures:

• Experience Replay: Reusing outdated transitions often induces off-policy errors where gradients conflict with new tasks, destabilizing learning.
• Second-order Regularization: Methods like Fisher information are computationally prohibitive for high-dimensional Transformers.
• Lightweight Constraints: Approaches like KL regularization often enforce only short-horizon behavioral similarity, failing to preserve long-term task performance.
• Backbone Freezing: Assuming pre-trained representations remain aligned with language goals is fragile; under task shifts, value learning drifts without explicit language conditioning, leading to catastrophic forgetting.

The core problem is identifying the mechanism driving forgetting in continual VLA learning and developing a framework that rigorously balances stability and plasticity without the computational or architectural overhead of existing methods.

Methodology: CRL-VLA

The authors propose CRL-VLA, a continual post-training framework grounded in a unified theoretical bound linking the stability-plasticity trade-off to goal-conditioned advantage magnitude.

Theoretical Foundation

The paper derives a unified performance bound showing that performance degradation on old tasks and improvement on new tasks are governed by the coupling of advantage magnitude ($M_g$) and policy divergence (measured by KL divergence).

• Stability ($M_{old}$): The maximum advantage of the old policy over its state distribution.
• Plasticity ($M_{new}$): The maximum advantage achievable on new tasks.

The authors demonstrate that $M_{old}$ and $M_{new}$ can be controlled independently:

1. Stability Control: $M_{old}$ is bounded by the critic approximation error ($\epsilon_V$) on old task data.
2. Plasticity Control: $M_{new}$ is naturally bounded by the environment's return range when using Monte Carlo (MC) estimation on new tasks.

This insight allows for asymmetric regulation: suppressing advantage magnitudes on prior tasks to ensure stability while permitting controlled growth on new tasks to enable plasticity.

Architecture: Dual-Critic with Goal-Conditioned Value Formulation (GCVF)

To operationalize this theory, CRL-VLA employs a dual-critic architecture:

1. Frozen Goal-Conditioned Value (GCV) Critic ($\theta_{GCV}$): Initialized from the converged value network of the previous task and frozen during new task training. It anchors semantic consistency and provides reference values ($V_{old}$) to constrain value approximation error on old data.
2. Trainable MC Critic ($\theta_{MC}$): Updated on new task trajectories using MC returns. It drives adaptation and allows the advantage magnitude on new tasks to remain naturally bounded by the environment's reward range.

Both critics share a unified backbone ($\phi$) but have decoupled heads. The value network is conditioned on language goals by concatenating language embeddings with state representations before the MLP, addressing poor language-goal following in standard VLA value heads.

Training Objectives

The total training loss integrates four components to balance the trade-off:
$$\mathcal{L}_{total} = \mathcal{L}_{PPO} + \alpha \mathcal{L}_{KL} + \beta_V \mathcal{L}^V_{GCV} + \eta \mathcal{L}^V_{MC}$$

• $\mathcal{L}_{PPO}$: Standard PPO objective for maximizing new task returns with a trust region constraint ($D_{new}$).
• $\mathcal{L}_{KL}$: KL divergence penalty on old task data to bound policy divergence ($D_{old}$).
• $\mathcal{L}^V_{GCV}$: Consistency loss between the frozen GCV critic and the current backbone's value predictions on old task data, minimizing $\epsilon_V$ to control $M_{old}$.
• $\mathcal{L}^V_{MC}$: Loss for the trainable MC critic on new task data, ensuring accurate advantage estimation bounded by the return range to control $M_{new}$.

Key Contributions

1. Theoretical Insight: The paper identifies the goal-conditioned advantage magnitude as the key quantity governing the stability-plasticity trade-off in continual VLA learning, deriving unified bounds that link this magnitude to policy divergence.
2. Asymmetric Regulation Framework: CRL-VLA operationalizes this insight via a dual-critic architecture that decouples stable value semantics (frozen GCV critic) from adaptive policy learning (trainable MC critic), enabling independent control of stability and plasticity.
3. Goal-Conditioned Value Formulation (GCVF): A novel formulation that conditions value networks directly on language embeddings, improving long-horizon value consistency and language-goal following.
4. Empirical Validation: Extensive experiments on the LIBERO benchmark demonstrate that CRL-VLA outperforms strong baselines (Sequence Learning, LWF, Experience Replay, Multi-Task Learning) in both anti-forgetting (Backward Transfer) and forward adaptation (Forward Transfer) metrics.

Experimental Results

The authors evaluated CRL-VLA on single-task and multi-task learning scenarios using the LIBERO benchmark with OpenVLA-OFT as the base model.

• Single-Task Learning: CRL-VLA (Q-based) achieved the highest Final Average Return (FAR) of 0.98, with the lowest forgetting metric (0.03) and a Backward Transfer (BWT) of -0.02, significantly outperforming baselines like Sequence Learning (FAR 0.00) and standard Experience Replay (FAR 0.60).
• Multi-Task Learning: CRL-VLA (V-based) achieved the highest FAR (0.74) and BWT (0.17), demonstrating superior cross-task knowledge sharing and resistance to catastrophic forgetting compared to MTL and ER baselines.
• Ablation Studies:
  • Loss Weights: Moderate weights for GCV and KL losses were found optimal; excessive weights constrained updates too much, while insufficient weights failed to suppress advantage magnitude.
  • GCV Necessity: Removing the GCV component led to a significant drop in success rates (FAR dropped from 0.31 to 0.29 in specific ablation settings) and increased forgetting, confirming the importance of language-conditioned value anchoring.
  • V vs. Q: The Q-based variant performed better in single-task scenarios, while the V-based variant showed robustness in multi-task environments where state-goal distributions are more diverse.

Significance and Claims

The paper claims that CRL-VLA provides a principled and effective pathway for deploying VLA models in lifelong robotic scenarios. By rigorously deriving theoretical bounds and translating them into a simple dual-critic architecture, the method resolves the stability-plasticity dilemma without requiring expensive second-order computations or large replay buffers.

The authors state that their work is the first to explicitly model the stability-plasticity trade-off in VLA continual learning via advantage magnitude control. They emphasize that CRL-VLA effectively harmonizes conflicting objectives, allowing agents to acquire new skills continuously while retaining prior knowledge, thereby advancing the feasibility of lifelong learning for embodied agents in open-world environments. Future work is suggested to extend this framework to more diverse task streams, including partial observability and non-stationary language goals.