Technical Summary: Z-1 – Efficient Reinforcement Learning for Vision-Language-Action Models

1. Problem Statement

Vision-Language-Action (VLA) models have emerged as a promising framework for robotic manipulation, unifying language instructions, visual observations, and continuous control. However, current state-of-the-art (SOTA) policies are predominantly trained via Behavior Cloning (BC) or Supervised Fine-Tuning (SFT) on fixed human demonstrations. This reliance on imitation learning presents two critical limitations:

1. Limited Generalization: Performance is bounded by the coverage of the demonstration data. In long-horizon or compositional tasks, small prediction errors can accumulate, pushing the robot into states poorly represented in the training data.
2. Inability to Self-Correct: BC and SFT lack mechanisms to improve from the policy's own failures. They cannot optimize for sparse task success through trial-and-error interaction, which is essential for refining complex manipulation behaviors.

While Reinforcement Learning (RL) post-training offers a pathway to overcome these limitations by optimizing task-level rewards, applying RL to flow-based VLA models remains challenging. Key obstacles include the high cost of rollout generation, the sparsity of rewards, and the difficulty of correcting failures that stem from visual grounding or spatial reasoning when the vision-language backbone is frozen.

2. Methodology: The Z-1 Framework

Z-1 is a modular RL post-training framework designed specifically for flow-based VLA models. Built upon the pretrained $\pi_0.5$ base model, Z-1 employs a two-stage training pipeline using only publicly released RoboCasa demonstrations.

Stage 1: Per-Scene Supervised Fine-Tuning (SFT)

The pipeline begins by adapting $\pi_0.5$ to the RoboCasa environment. Instead of training a separate model for every task, Z-1 performs per-scene SFT (grouping tasks by scene category, e.g., doors, drawers, sinks). This approach improves data efficiency by leveraging shared object layouts and manipulation primitives within a scene. Both the vision-language module and the action expert are updated during this stage to produce a robust initialization.

Stage 2: Task-Wise Group Relative Policy Optimization (GRPO)

Following SFT, Z-1 applies a task-specific GRPO post-training strategy. Unlike standard RL which treats rollouts as independent full trajectories, Z-1 introduces four key modular components to enhance efficiency and stability:

• Shared-Prefix GRPO: To address the issue where early "approach" phases vary significantly and dilute reward signals, Z-1 forces rollouts within a GRPO group to share a common prefix. The simulator state at the branching point is cloned, and only the suffixes (post-branching actions) are sampled independently. This ensures that group-relative comparisons focus on task-critical interaction phases rather than redundant approach variations.
• Tree-Structured Prefix Branching: For tasks requiring refinement during the approach phase, Z-1 generalizes the flat shared-prefix approach. It introduces intermediate branching points within the prefix, creating a tree of rollout clusters. This reduces over-masking, allowing partially shared segments to contribute to the learning signal while still reducing redundant computation.
• Success-Aware Reward Decay: To handle sparse rewards where all successful trajectories are treated equally, Z-1 introduces a decay factor based on completion time ($\tilde{r}_i = r \cdot \gamma^{d_i}$). Faster successful completions receive higher rewards. Crucially, for groups where all rollouts succeed, Z-1 uses a non-negative minimum-baseline ordering ($A_i = R_i - \min_j R_j$) rather than mean normalization. This prevents slower successful rollouts from receiving negative advantages, avoiding the over-penalization of reliable but slower completions.
• Selective VLM–Action Expert Joint Training (Sel-JT): While standard VLA-RL often freezes the vision-language backbone, Z-1 dynamically selects whether to update the Vision-Language Model (VLM) alongside the Action Expert. This decision is based on training-stage diagnostics (e.g., SFT success rates, failure modes). If failures are attributed to poor visual grounding or spatial reasoning rather than low-level control, the VLM is unfrozen to enable perception-action co-adaptation.

3. Key Contributions

The paper presents the following contributions:

1. Z-1 Framework: A modular GRPO post-training framework for flow-based VLA models that operates effectively with only public demonstration data, avoiding the need for private datasets.
2. Novel Rollout Construction: The proposal of Shared-Prefix GRPO and Tree-Structured Prefix Branching to reduce redundant simulation and focus optimization on critical interaction segments.
3. Reward and Training Innovations: The development of Success-Aware Reward Decay to provide finer feedback on completion speed without dense reward shaping, and Selective Joint Training to adapt perception modules only when necessary.
4. New SOTA Performance: The establishment of a new state-of-the-art average success rate on the RoboCasa benchmark under public demonstration settings.

4. Experimental Results

Z-1 was evaluated on 24 standard RoboCasa manipulation tasks across 7 categories (doors, drawers, pick-and-place, microwave, sink/faucet, stove, and coffee-making).

• Performance Gain: Z-1 RL improved the average success rate from 67.4% (SFT initialization) to 80.6%, a gain of 13.2 percentage points.
• Comparison with SOTA: Z-1 RL outperformed previously published results, including GR00T, GR00T N1.5, Video Policy, and X-WAM. Specifically, it surpassed X-WAM (the strongest baseline in the comparison) by 1.4 percentage points on average, despite X-WAM utilizing depth information while Z-1 relied solely on public demonstrations.
• Task-Specific Improvements:
  • Drawer & Sink/Faucet: Significant improvements were observed (e.g., Drawer: 83.4% $\to$ 96.1%; Sink: 63.2% $\to$ 94.3%), demonstrating the efficacy of prefix-based rollout construction for tasks requiring precise post-approach manipulation.
  • Pick-and-Place: Substantial gains were achieved (53.9% $\to$ 70.9%).
  • Stove Tasks: While Z-1 improved over its SFT baseline, it remained slightly below X-WAM, indicating that tasks requiring complex long-horizon transport and robust recovery still present challenges.
• Ablation Studies:
  • Prefix-based variants (Shared-Prefix and Tree-Structured) consistently outperformed vanilla GRPO, with Tree-Structured offering higher peak performance when training remained stable.
  • Selective Joint Training (Sel-JT) significantly outperformed Action-Expert-only training on difficult tasks like "TurnOnStove," validating the need to adapt visual representations when grounding is insufficient.
  • Success-Aware Reward Decay provided faster and more stable success-rate improvements compared to standard reward penalties, which tended to be overly aggressive.

5. Significance and Claims

The paper claims that systematic GRPO post-training can substantially improve flow-based VLA policies without relying on additional private demonstrations.

The authors emphasize that Z-1 demonstrates the viability of online RL for VLA models by addressing specific inefficiencies in rollout generation and reward calibration. The results suggest that with careful design—specifically through shared-prefix construction, completion-aware reward calibration, and adaptive module selection—RL can effectively refine policies beyond the limits of imitation learning. The work establishes a new benchmark for public-data-only VLA training, showing that algorithmic improvements in post-training can yield performance gains comparable to or exceeding models trained with more extensive or proprietary data.