🤖 machine learning

Reinforcement Learning for Real-Time Vision-Language-Action Policies

This paper introduces Real-Time EXPO-FT, a reinforcement learning framework that decouples slow, expressive action generation from fast, reactive action editing to enable sample-efficient, high-performance adaptation of large Vision-Language-Action models to real-world dynamics despite inference latency.

Original authors: Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn

Published 2026-09-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to move with the fluidity of living things. While they can be programmed to follow a rigid set of instructions in a factory, the real world is messy, unpredictable, and moves fast. To navigate this, researchers have turned to a type of artificial intelligence known as a vision-language-action model. These are massive computer programs trained on huge amounts of data, allowing them to understand what they see through a camera, read a text instruction, and decide how to move a robotic arm. They are powerful because they have seen so many examples of how the world works, acting like a vast library of experience. However, there is a significant catch: because these models are so large and complex, they take a noticeable amount of time to think. In the split second it takes the computer to process an image and decide on a movement, the physical world has already changed. If a robot is trying to catch a ball or balance an object, the information it used to make its decision is already old by the time it actually moves, often causing the action to fail.

A team of researchers at Stanford University has developed a new way to teach these large models how to act in real time, even when they are slow to think. Their approach, called Real-Time EXPO-FT, solves the problem of delay by splitting the robot's decision-making into two distinct parts. Instead of forcing the massive, slow model to make every single split-second adjustment, they let it do the heavy lifting of planning a general path, while a much smaller, faster computer program handles the immediate corrections. Imagine a conductor leading an orchestra; the conductor sets the overall tempo and melody, but the individual musicians must adjust their playing instantly to stay in sync. In this system, the large model acts as the conductor, proposing a sequence of moves based on what it saw a moment ago. Then, a lightweight assistant watches the current state of the world and makes tiny, rapid edits to those proposed moves to ensure they are still correct for the exact moment the robot needs to act. This allows the robot to use the deep knowledge of the large model without being held back by its slow thinking speed.

The researchers tested this method in two very different environments: a simulated world of physics and the actual physical world. In the simulation, they challenged the robot with ten different dynamic tasks, such as balancing a ball on a plate, kicking a soccer ball while avoiding a defender, and catching a moving object. They compared their new method against several existing techniques that tried to handle delays. The results were clear: the new approach allowed the robot to succeed in nearly every single trial, outperforming all other methods, including those that did not have to deal with delays at all. This was a significant finding because it showed that by explicitly teaching the system to account for its own slowness, the robot could become more reliable than systems that were theoretically faster but less adaptable.

To prove this worked in the real world, the team moved the robot to a laboratory setting and gave it four challenging tasks that required constant, rapid reaction. These included keeping a ping-pong ball balanced on a rotating plate, picking up a block from a spinning surface, catching an object passed by another robot arm, and kicking a ball into a goal while a defender moved around. The researchers limited the training time to just ten minutes of the robot interacting with the environment, ensuring the system could learn quickly without needing endless hours of practice. Before applying their method, the robot's success rate on these tasks was quite low, averaging around 42 percent. After using the new real-time training framework, the success rate jumped to 97 percent. The robot learned to adapt to the chaotic movements of the objects and the timing constraints of the tasks without any human intervention during the training process.

The study also explored how well the system held up under different conditions. When the researchers artificially increased the delay in the robot's thinking process, the new method maintained its high level of performance, whereas other methods failed as the delay grew longer. Similarly, when the speed of the moving objects increased, the robot using this new framework continued to succeed, while other approaches struggled to keep up. This demonstrates that the system does not just work for a specific speed or delay but is robust enough to handle the unpredictable nature of a dynamic environment. The researchers noted that while their system is highly effective, it still relies on a human to reset the robot after a task is finished, and it requires a specific way of defining success for each task. However, the core achievement is a demonstration that large, powerful AI models can be made to work in real-time, bridging the gap between the slow, thoughtful planning of artificial intelligence and the fast, reactive demands of the physical world.

Technical Summary: Real-Time EXPO-FT

Problem Statement
Reinforcement learning (RL) fine-tuning of large, pretrained Vision-Language-Action (VLA) models holds promise for reliable robot deployment. However, the scale of these models introduces significant inference latency. In dynamic real-world environments, the observation used to generate an action is often stale by the time execution occurs. This creates a distribution shift where the policy acts on outdated state information, substantially degrading reliability and performance. While prior work has explored asynchronous execution (e.g., Real-Time Chunking) to mitigate latency, these methods are primarily built on imitation learning and lack mechanisms to move beyond the training distribution. Furthermore, directly applying standard RL to delayed policies fails to capture the full reliability gains of RL because actions are predicted from previous observations, breaking the Markov property required for standard credit assignment.

Methodology: Real-Time EXPO-FT
The paper proposes Real-Time EXPO-FT, a framework that decouples slow, expressive action generation from fast, reactive action editing to enable RL fine-tuning under real-time constraints. The approach builds upon EXPO-FT, a sample-efficient RL system for VLA fine-tuning, and integrates it with Training-Time Real-Time Chunking (RTC) concepts.

The core architecture consists of three components operating on different timescales:

Asynchronous VLA Action Generation (Slow): A large, pretrained VLA base policy ( $\pi_{VLA}$ ) runs asynchronously in the background. It generates multiple candidate action chunks based on an observation $s_t$ and the "committed" action prefix from the previous chunk ( $a^{prev}_{t:t+d}$ ) to account for the inference delay $d$ . This step utilizes flow-matching to sample diverse candidates.
Fast, Synchronous Edits (Fast): Once the robot reaches the execution time $t+d$ and acquires the latest observation $s_{t+d}$ , a lightweight edit policy ( $\pi_{edit}$ ) performs reactive decision-making. It takes the candidate chunks generated by the VLA and edits them based on the current state to correct for any state changes that occurred during the inference delay.
Selection via Q-Function: A learned Q-function ( $Q_\phi$ ) evaluates both the original VLA candidates and the edited candidates under the current state $s_{t+d}$ . The system selects the action chunk with the highest Q-value for execution.

Training Procedure

VLA Fine-tuning: The base VLA is fine-tuned using a modified flow-matching objective that simulates inference delay during training. The loss is masked to exclude the first $d$ actions (the delay window), forcing the model to learn to predict the remaining actions conditioned on the committed prefix.
Edit Policy Training: The edit policy is trained to maximize the Q-value of the edited actions, effectively learning to shift the base policy's output toward higher-value regions of the action space.
Critic Training: The critic is trained using temporal-difference learning over full action chunks. To reduce computational cost during the Bellman backup, the method employs a noise-space filtering mechanism (inspired by FASTER). A lightweight critic ( $Q^{dn}_\psi$ ) scores candidate noise vectors before full decoding, allowing the system to filter candidates in noise space and only perform a single VLA decode for the best candidate before applying edits.

Key Contributions

Framework for Real-Time RL: The paper introduces the first framework enabling RL fine-tuning of large VLA models that explicitly accounts for inference latency while maintaining real-time control frequencies.
Decoupled Control Architecture: It proposes a novel decoupling of slow, high-capacity behavior generation (VLA) and fast, reactive state correction (Edit Policy), allowing the system to leverage the VLA's prior knowledge while retaining reactivity.
Sample-Efficient Adaptation: By combining EXPO-FT's sample efficiency with real-time execution strategies, the method achieves rapid adaptation to dynamic environments with minimal online data.

Experimental Results
The authors evaluate Real-Time EXPO-FT on the Kinetix benchmark (10 simulation tasks) and four dynamic real-world tasks (Robot Object Passing, Ball Balancing, Table Soccer Kicking, and Dynamic Object Picking).

Simulation: On the Kinetix benchmark, Real-Time EXPO-FT achieved the best performance among both delayed and non-delayed methods in 10 out of 10 environments. It significantly outperformed delayed baselines (DSRL, EXPO-FT with/without RTC) and even surpassed the non-delayed RLPD baseline (81.4% vs. 96.2% average success rate) despite operating with a 4-step inference delay.
Real-World: In real-world experiments capped at 10 minutes of online robot data, Real-Time EXPO-FT improved the average policy success rate from 42% (12.5/30) to 97% (29/30). It achieved near-perfect performance (30/30) on Dynamic Picking and Object Passing, and 28/30 on Ball Balancing and Soccer Kicking, substantially outperforming all baselines.
Robustness: The method demonstrated stable performance across varying inference delays and environment speeds, whereas baselines like RTC degraded significantly as delays increased.

Significance and Claims
The paper claims that Real-Time EXPO-FT successfully bridges the gap between the high reliability of RL fine-tuning and the reactivity required for dynamic real-world manipulation. By explicitly modeling inference latency and decoupling action generation from execution, the method allows large VLA models to be deployed in high-frequency control loops without sacrificing performance. The results demonstrate that it is possible to achieve rapid, sample-efficient adaptation to challenging real-world dynamics without human intervention during training, overcoming the distribution shift caused by model latency.

Limitations
The authors acknowledge that their current implementation relies on human operators for environment resets, which introduces operational burden. Additionally, the reward signal relies on task-specific success classifiers (rule-based or learned), requiring manual design for each new task. The paper notes that automating resets and exploring alternative reward formulations remain open questions for future work.

Technical Summary: Real-Time EXPO-FT

More like this