Technical Summary: DyGRO-VLA

1. Problem Statement

The paper addresses a critical scalability bottleneck in training Vision-Language-Action (VLA) models for generalist robot manipulation. While recent advances in Reinforcement Learning (RL) have improved control precision and success rates for individual tasks, applying RL post-training to multi-task settings often leads to catastrophic forgetting and cross-task interference.

Empirical analysis on the LIBERO benchmark reveals that as the number of tasks increases, standard RL optimizers (e.g., Soft Actor-Critic) degrade in performance. Specifically:

• Representation Distortion: Single-task RL fine-tuning distorts the shared feature space learned during pre-training, isolating task-specific experiences and reducing alignment with other tasks.
• Performance Degradation: Models optimized for specific tasks often suffer significant performance drops on unrelated tasks, and multi-task RL training becomes increasingly unstable as the task count grows.
• The Trade-off: There is a fundamental conflict between achieving high dexterity on specific tasks and maintaining the cross-task generalizability required for generalist robots.

2. Methodology: DyGRO-VLA

To resolve the trade-off between dexterity and generalizability, the authors propose DyGRO-VLA (Dynamic Grouped Residual Optimization), a two-stage optimization framework.

Stage 1: Offline Pre-training with Information Bottleneck (IB)

The first stage focuses on learning a robust, shared cross-task latent representation $p(z|o)$ from offline demonstrations.

• Architecture: Utilizes a VLM backbone (DINOv2, SigLIP, Qwen2.5-0.5B) to encode visual observations and language instructions, fused with proprioceptive data.
• Objective: The model is trained using a Variational Information Bottleneck (VIB) objective. The goal is to maximize the mutual information between the latent representation $Z$ and the action $A$ ($I(Z; A)$) while minimizing the mutual information between the observation $O$ and $Z$ ($I(O; Z)$).
• Mechanism: This forces the encoder to retain only task-relevant information (e.g., object geometry, spatial relations) while discarding nuisance factors (e.g., background clutter, lighting), creating a compact latent space suitable for generalization.
• Loss: A variational lower bound is derived to make the IB objective tractable, involving a neural critic $T_\psi$ to estimate the mutual information terms.

Stage 2: Online Finetuning via Mixture-of-RL-Residuals (MoRR)

The second stage optimizes the policy for multi-task performance without disrupting the shared representation learned in Stage 1.

• Residual Learning: Instead of directly updating the base policy, the RL optimizer predicts a residual ($\Delta a$) to refine the base action chunk ($a_{base}$). This preserves the pre-trained prior while allowing for task-specific corrections.
• Dynamic Routing (MoE): To handle multiple tasks, the framework employs a Mixture-of-Experts (MoE) architecture with $N$ residual experts.
  • Task Embeddings: A contrastive learning objective ($L_{CL}$) learns task embeddings ($\tilde{z}_T$) that encode task-identification patterns, distinct from the decision-relevant latent $z$.
  • Router: A gating network conditions on the task embedding to dynamically select and weight the top-$m$ experts for a given task.
• Training Objectives: The total loss combines:
  1. RL Loss ($L_{RL}$): Standard actor-critic loss for the augmented policy.
  2. Contrastive Loss ($L_{CL}$): Ensures task embeddings are discriminative for routing.
  3. Load Balancing ($L_{LB}$): An entropy-based regularizer to prevent expert collapse and ensure balanced utilization of the MoE.
• Difficulty-Aware Sampling: The training process biases sampling toward under-solved tasks to focus learning on challenging scenarios.

3. Key Contributions

1. Analysis of RL Limitations: The paper provides empirical evidence demonstrating that standard RL post-training causes catastrophic forgetting and representation distortion in multi-task VLA settings, highlighting the need for cross-task feature preservation.
2. DyGRO-VLA Framework: Introduction of a novel two-stage framework that combines Information Bottleneck pre-training with a dynamic Mixture-of-RL-Residuals for online fine-tuning.
3. Dynamic Grouped Optimization: The design of a routing mechanism that dynamically composes specialized RL experts based on task embeddings, effectively mitigating gradient conflicts between diverse tasks.
4. Comprehensive Validation: Extensive evaluation across LIBERO (130 tasks), RoboTwin2 (dual-arm), and real-world Sim2Real transfer, demonstrating consistent improvements over state-of-the-art baselines.

4. Experimental Results

The authors evaluated DyGRO-VLA on the LIBERO and RoboTwin2 benchmarks, comparing it against strong baselines including Octo, OpenVLA, $\pi_0$, and various RL fine-tuning methods.

• LIBERO Benchmark:
  • DyGRO-VLA achieved a 97.1% average success rate across all suites, outperforming the offline base model by +4.4%.
  • Notable improvement was observed on the challenging LIBERO-Long suite, where success rates increased from 85.2% to 95.0% (+9.8% absolute gain).
  • It outperformed other RL-based baselines (e.g., TGRPO, VLA-RL, RIPT-VLA) which struggled with multi-task co-training.
• Gradient Conflict Analysis: Visualizations of gradient cosine similarity showed that while standard MLP residuals exhibit strong negative correlations (conflicts) between task groups, DyGRO-VLA's MoRR significantly reduced these conflicts while maintaining intra-group synergy.
• Ablation Studies:
  • Removing the IB objective or the contrastive learning objective led to performance drops, confirming the necessity of shared representation learning and discriminative routing.
  • Increasing the number of experts (from 1 to 8) consistently improved performance, with 8 experts yielding the best results.
• Real-World (RoboTwin2):
  • In Sim2Real transfer, DyGRO-VLA achieved the best overall simulation success (79.2%) and surpassed RFT baselines in real-world trials, particularly on complex bimanual and long-horizon tasks.

5. Significance and Claims

The paper claims that DyGRO-VLA offers a scalable and robust solution for efficient multi-task reinforcement fine-tuning. Its primary significance lies in:

• Preserving Generalization: By decoupling the learning of shared representations from task-specific refinement via residual learning, it prevents the "overfitting" of RL optimizers that typically plagues multi-task VLA training.
• Bridging the Gap: It successfully bridges the gap between high-performance single-task control and the generalization capabilities required for generalist robots.
• Practical Applicability: The method demonstrates that complex, multi-task RL policies can be effectively transferred to real-world robotic systems without catastrophic forgetting.

The authors acknowledge limitations, including sensitivity to reward sparsity during online training, reliance on known task identities for routing, and the need for broader real-world evaluation across diverse embodiments. However, they position DyGRO-VLA as a foundational step toward general-purpose robotic systems capable of handling diverse real-world tasks with a unified model.