💻 computer science

RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models

The paper introduces RLinf-VLA, a unified and efficient framework that standardizes the integration of diverse VLA architectures and RL algorithms while optimizing resource allocation to achieve significant training speedups and state-of-the-art performance across multiple embodied intelligence benchmarks.

Original authors: Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, Hao Lin, Peihong Wang, Liangzhi Shi, Yuqing Xie, Zhexuan Xu, Zhihao Liu, Kang Chen, Wenhao Tang, Quanlu Zhang, Weinan Zhang, Chao

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, Hao Lin, Peihong Wang, Liangzhi Shi, Yuqing Xie, Zhexuan Xu, Zhihao Liu, Kang Chen, Wenhao Tang, Quanlu Zhang, Weinan Zhang, Chao Yu, Yu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just mindless machines following a strict list of instructions, but curious learners that can understand what you say, see what you see, and figure out how to move their own bodies to get things done. This is the exciting frontier of Vision-Language-Action (VLA) models. Think of these models as the robot's "brain," trained on massive amounts of internet data to understand language and images. But there's a catch: just because a robot knows what a cup is and hears "pick up the cup," doesn't mean it knows the best way to actually grab it without knocking it over. To get really good at physical tasks, these robots need to practice, make mistakes, and learn from the results. This is where Reinforcement Learning (RL) comes in. You can think of RL as a video game training loop: the robot tries an action, gets a "score" (reward) if it succeeds, and learns to repeat the good moves while avoiding the bad ones. The big question scientists are asking is: how do we build a training system fast enough and smart enough to let these robots learn complex skills without taking forever?

Enter RLinf-VLA, a new framework that acts like a super-efficient training gym for these robot brains. The researchers behind this paper realized that while we have powerful robot models and great training algorithms, the "gym" itself was often broken. Previous systems were like trying to run a marathon while carrying a heavy backpack; they were slow, clunky, and couldn't handle different types of training environments well. Sometimes the robot's brain (the model) was waiting for the simulator (the virtual world) to catch up, and other times the simulator was waiting for the brain, leaving expensive computer chips sitting idle.

The team built RLinf-VLA to fix this traffic jam. They created a unified system that can plug in different robot simulators, different brain architectures, and different learning algorithms all at once. But the real magic is in how they manage the computer power. They introduced a clever "hybrid" mode that acts like a well-choreographed dance. Instead of letting the robot's brain and the virtual world wait for each other, they pipeline the process: while the brain is thinking about the next move for one part of the simulation, the simulator is already running the previous move for another part. This keeps everything moving at full speed.

The results of this new system are impressive. In their simulations, the framework made training up to 2.27 times faster than previous methods. When they put their trained models to the test, the results were strong. A single model trained with RLinf-VLA achieved a 98.11% success rate across 130 different tasks in the LIBERO benchmark and 97.66% on 25 tasks in ManiSkill. Even on the tricky RoboTwin tasks, which involve using two hands, the models reached an average success rate of 84.63%. The researchers also tested a version of this in the real world with a physical robot arm closing a drawer. While the success rate was the same as a standard model (8 out of 10 tries), the RL-trained robot moved more smoothly and efficiently, avoiding the awkward, jerky movements of the untrained version.

The paper suggests that this new framework isn't just a speed boost; it's a foundational tool that makes it possible to train these complex robot brains on a much larger scale than before. By standardizing how these systems talk to each other and optimizing how they use computer resources, RLinf-VLA offers a path toward robots that can learn new physical skills quickly and reliably, bringing us one step closer to the day when robots can truly help us in our daily lives.

Technical Summary: RLinf-VLA

Problem Statement

Recent advancements in Vision-Language-Action (VLA) models have demonstrated the potential of reinforcement learning (RL) to enhance task performance through interaction. However, current research in this domain faces three primary challenges:

Fragmentation: Existing efforts lack a unified platform for fair comparison across diverse VLA architectures, RL algorithms, and simulators.
Inefficiency: Scalable RL training is hindered by system designs that do not account for the specific resource contention in embodied settings. Unlike Large Language Models (LLMs), VLA training involves tight coupling between model inference, simulator rendering, and optimization, often leading to significant GPU idle time and pipeline bubbles.
Lack of Standardization: The absence of a unified interface makes it difficult to extract general principles for RL-based VLA training, as existing works rely on disparate protocols and codebases.

Methodology

The authors present RLinf-VLA, a unified and efficient framework designed to address these challenges through system-level abstraction and algorithmic optimization.

1. Unified System Abstraction

RLinf-VLA provides a standardized interface that integrates:

Multiple Simulators: Supports heterogeneous environments including CPU-parallelized (e.g., LIBERO) and GPU-parallelized (e.g., ManiSkill, RoboTwin) simulators.
Diverse Architectures: Accommodates various VLA models (e.g., OpenVLA, OpenVLA-OFT, $\pi_0$ ) and action formulations (single-step vs. action chunks).
RL Algorithms: Supports on-policy algorithms like PPO and GRPO, with extensibility for off-policy methods.

2. Flexible GPU Allocation Strategies

To maximize resource utilization, the framework introduces three execution modes:

Collocated Mode: All components (Generation, Simulator, Training) share the same GPUs. The authors modified the standard approach to minimize offload-onload overhead by restricting these operations to the start and end of rollout phases.
Disaggregated Mode: Components are assigned to distinct GPU partitions. While simple, this often leads to underutilization due to inter-component dependencies (e.g., GPUs waiting for data).
Hybrid Mode (Novel): This mode combines the benefits of the previous two. It assigns Generation and Simulator to different GPU partitions while allowing Training to utilize all GPUs. Crucially, it introduces fine-grained pipelining, where simulator instances are partitioned into sub-simulators ( $S^{(1)}, S^{(2)}, \dots$ ). This allows the Simulator and Generation components to run concurrently, mitigating "GPU bubbles" caused by sequential dependencies without incurring the overhead of frequent data offloading.
Auto-Placement: A lightweight algorithm automatically determines the optimal GPU allocation and pipeline depth ( $k=1$ or $k=2$ ) based on runtime profiling to minimize rollout time.

3. Algorithmic Design Choices

The framework distills specific design choices for scaling RL to VLAs:

Multi-Granularity Support: Supports advantage and log-probability computation at Chunk, Action, and Token levels.
PPO Enhancements:
- Partial Reset: Resets sub-environments immediately upon termination rather than waiting for a fixed episode length, improving sample efficiency.
- Parameter-Sharing Critic: Attaches a lightweight value head to the actor's language model to avoid the computational cost of a separate critic network.
- Action-Level Value Estimation: Empirically found to outperform chunk-level estimation for action chunks.
GRPO Enhancements:
- Valid Action Mask: Masks out timesteps occurring after task completion to focus optimization on relevant steps.
- Trajectory Length Normalization: Normalizes loss by the number of valid timesteps to prevent longer trajectories from dominating gradients.
- Success Rate Filter: Discards trajectory groups where all outcomes are identical (all success or all failure) to ensure meaningful advantage calculation.

Key Results

The framework was evaluated across three benchmarks: LIBERO (130 tasks), ManiSkill (25 tasks), and RoboTwin (6 tasks).

Performance Gains:
- LIBERO: A single unified model achieved a 98.11% success rate across 130 tasks (up from ~42% baseline).
- ManiSkill: A single model achieved 97.66% success on 25 tasks (up from ~18% baseline for OpenVLA-OFT).
- RoboTwin: Six task-specific models achieved an average success rate of 84.63% (a 63.75% improvement over the base model).
- RL-trained models consistently outperformed Supervised Fine-Tuning (SFT) baselines, particularly in Out-of-Distribution (OOD) generalization.
System Efficiency:
- The Hybrid mode with fine-grained pipelining achieved speedups of 1.61×–1.88× on ManiSkill compared to disaggregated baselines.
- Overall, RLinf-VLA achieved up to a 2.27× speedup over existing frameworks (e.g., SimpleVLA-RL) by optimizing both rollout and training phases.
Real-World Transfer:
- A preliminary real-world experiment on a drawer-closing task showed that an RL-optimized policy completed the task more efficiently than an SFT-only baseline, exhibiting more direct and fluent behaviors despite being trained primarily in simulation.

Significance and Claims

The authors position RLinf-VLA as a foundational framework for the field of embodied intelligence. Its significance lies in:

Unification: It provides the first unified interface to support diverse simulators, models, and algorithms, enabling fair comparison and reproducibility.
Efficiency: By addressing the specific resource contention issues of embodied RL (simulator vs. inference vs. training), it offers a scalable solution that significantly reduces training time.
Best Practices: The paper distills actionable design choices (e.g., action-level value estimation, partial reset, valid action masking) that guide future research in RL-based VLA training.
Openness: Released as an open-source, actively maintained platform, it aims to accelerate and standardize research in this domain.

The paper concludes that while simulation-based RL shows promise for real-world deployment, the primary contribution is the establishment of a robust, efficient, and extensible infrastructure necessary to advance VLA capabilities through reinforcement learning.