Technical Summary: EXPO-FT

Problem Statement

The ability to efficiently and reliably learn new tasks remains a foundational challenge in robotics. While Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks through large-scale imitation learning, pretrained policies consistently fall short of the reliability required for real-world deployment. Failures in physical environments are costly, necessitating a path to bridge the gap between generalist pretraining and reliable, task-specific performance.

Existing approaches to training real-world robot policies via Reinforcement Learning (RL) face a dichotomy:

1. Training from scratch: Methods that train RL policies without finetuning a VLA (or train lightweight models on top) often achieve strong asymptotic performance but fail to leverage the rich semantic and behavioral priors embedded in large pretrained VLAs.
2. Finetuning VLAs: Methods that finetune pretrained VLAs inherit generalizable representations but often fail to reach reliable success rates, require prohibitively large numbers of environment interactions, or both.

There is a critical gap for frameworks that simultaneously exploit the prior of a pretrained VLA and achieve the sample efficiency and reliability necessary for practical deployment.

Methodology: EXPO-FT

The authors present EXPO-FT, a system designed for stable, sample-efficient RL finetuning of pretrained VLA policies. The system is built upon the EXPO algorithm, a recently proposed method for stable RL with expressive flow or diffusion policies, and extends it to address specific challenges in robotics.

Core Architecture and Design Choices

EXPO-FT integrates three primary components to achieve high sample efficiency and reliability:

1. Base Policy and Edit Mechanism (Extended EXPO):

   • The system utilizes a pretrained VLA (specifically $\pi_{0.5}$) as a base flow policy ($\pi_{VLA}$).
   • Instead of backpropagating gradients through the large VLA model during RL updates, EXPO-FT employs a lightweight edit policy ($\pi_{edit}$).
   • The edit policy predicts a correction $\hat{a}$ constrained to a small range $[-\beta, \beta]$, which is added to the base action $a$ to produce the edited action $\tilde{a} = a + \hat{a}$.
   • A Q-function is trained via Temporal Difference (TD) learning to select the value-maximizing candidate between the base action and the edited action. This allows the system to ground TD updates in near-optimal actions without destabilizing the pretrained VLA.

2. Temporally Extended Actions (Action Chunks):

   • Unlike classical RL algorithms designed for single-step Gaussian actions, modern VLAs operate over action chunks (sequences of $H$ future actions).
   • EXPO-FT adapts the edit policy and Q-function to operate natively over these chunks ($a_{t:t+C}$). The edit policy predicts edits for the entire chunk, preserving the temporal abstraction central to VLA design.

3. Human-in-the-Loop (HIL) Interventions:

   • To further accelerate learning and provide targeted corrections where autonomous exploration is insufficient, the system incorporates human-in-the-loop feedback.
   • During online training, a human operator can intervene via teleoperation (using a SpaceMouse) to provide action corrections for specific timesteps within an action chunk.
   • These corrective actions overwrite the policy's actions for the intervened timesteps, and the resulting trajectories are added to the replay buffer. This mechanism provides high-quality signals to reduce the exploration burden.

Training Procedure

The system employs a decoupled Learner-Actor architecture to manage the latency of large VLA inference and training:

• Actor: Executes rollouts in the real-world environment, collects human interventions, and sends state-action-reward tuples $(s, a, r, s')$ to the learner.
• Learner: Handles VLA training, inference, and maintains the replay buffer. It updates the agent (Q-network and edit policy) per step, per episode, or per large batch.
• Initialization: Before online RL, the VLA is often bootstrapped with a small offline dataset of human demonstrations via supervised finetuning (Imitation Learning) to reach a baseline success rate (e.g., ~40%).
• Reward: The system uses a sparse binary reward signal indicating task completion, avoiding the engineering overhead of dense reward shaping.

Key Contributions

1. System for Sample-Efficient RL Finetuning: A complete framework (EXPO-FT) that enables the reliable finetuning of pretrained VLA models using RL, achieving high success rates with minimal online data.
2. Architectural Extensions: The extension of the EXPO algorithm to support temporally extended action chunks and the integration of human-in-the-loop interventions specifically for VLA finetuning.
3. Open-Source Release: The release of an open-source codebase to facilitate broader adoption of RL finetuning for VLA models in the robotics community.

Experimental Results

The authors evaluated EXPO-FT on eight challenging real-world manipulation tasks requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Tasks included:

• Dynamic/Contact-rich: Egg Flip (flipping a fried egg with a spatula), Pool Shot (striking a billiard ball).
• Precision/Alignment: Flower Insert (inserting a stem into a bottle), Cube Pick (grasping under randomization), String Light Routing (hanging bulbs and plugging in power).
• Deformable/Noisy: Candy Scoop (transferring candies with a spoon).

Performance Metrics:

• Success Rate: EXPO-FT achieved a perfect success rate (30/30) across all evaluated tasks.
• Sample Efficiency: The system reached this reliable performance within an average of 19.1 minutes of online robot data.
• Comparison: EXPO-FT outperformed:
  • HIL-SERL (RL from scratch with HIL): Achieved only 5.5/30 average success, struggling with large initial state distributions.
  • DSRL (Latent space RL on VLAs): Achieved 19/30 average, limited by optimization constraints within the prior distribution.
  • HG-DAgger (Imitation learning with corrections): Achieved 22.1/30 average, struggling to recover from compounding errors in dynamic tasks.
  • Supervised Finetuning (SFT): Achieved 20.5/30 average.

Notably, on tasks requiring dynamic actions (Egg Flip, Pool Shot) or high precision, the performance gap between EXPO-FT and prior methods was most pronounced.

Significance and Claims

The paper claims that EXPO-FT demonstrates that RL finetuning of VLAs can be both sample-efficient and reliable when designed with the correct architectural choices. By combining the semantic priors of a pretrained VLA with an efficient off-policy RL algorithm (EXPO) and human-in-the-loop corrections, the system bridges the gap between generalist pretraining and the reliability required for real-world deployment.

The authors position this work as a stepping stone toward the broader adoption of finetuning generalist VLA models with reinforcement learning. They acknowledge limitations, such as the current requirement for manual environment resets and the computational challenges of running high-frequency training on billion-parameter models, identifying these as directions for future work. However, the core contribution is the validation that a unified, sample-efficient pipeline can transform a generalist VLA into a highly reliable specialist for complex manipulation tasks.