💻 computer science

SmoothRL: Online Reinforcement Learning During Asynchronous Execution

SmoothRL is an online reinforcement learning framework that enables the sample-efficient fine-tuning of pretrained robot policies within asynchronous inference loops by explicitly modeling the execution timeline and propagating gradients only through the newly executed action regions to ensure real-time reliability and smooth control.

Original authors: Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang

Published 2026-09-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to move with the fluid grace of living things. While artificial intelligence has taught machines to understand language and recognize objects with startling accuracy, translating that understanding into physical motion remains a stubborn hurdle. The problem is not just that robots must be smart; they must also be fast and smooth. In the real world, a robot arm cannot pause to think between every tiny movement. If it stops to calculate its next step, the motion becomes jerky, and the task fails. To move smoothly, a robot must predict a series of future movements and execute them while simultaneously calculating the next set. This creates a complex timing problem: the robot is acting on instructions that were generated a moment ago, while the computer is already working on the instructions for the moment after. If the robot tries to learn from its mistakes while operating under these conditions, the learning process often breaks down, because the robot is being judged on actions it never actually finished, or actions it was forced to abandon mid-stream.

A team of researchers at Astribot has developed a new method called SmoothRL to solve this specific timing puzzle. Their work allows robots to learn from experience in real time, even while they are running on a complex, overlapping schedule where thinking and moving happen at the same time. The researchers focused on a fundamental mismatch in how robots are trained versus how they are used. In the past, scientists trained robots by assuming the machine would stop, think, and then move in a perfect, synchronized rhythm. But in the real world, to keep motion smooth, robots use a system where they generate a "chunk" of future movements, send the first part to the motors, and immediately start calculating the next chunk while the first one is still being executed. This creates a situation where the robot is constantly discarding parts of its own plans because a newer, fresher plan has arrived. The researchers realized that for a robot to learn effectively in this environment, it must stop trying to learn from the parts of its plan that were thrown away and focus only on the parts it actually performed.

The core of their solution is a way to teach the robot to ignore the "ghost" actions it generated but never used. When the robot creates a plan for the next few seconds, it divides that plan into three distinct zones. The first zone contains actions that were already decided by the previous calculation cycle; the robot cannot change these now. The second zone contains the new actions that the robot actually performs while the next calculation is happening. The third zone contains the remaining future actions that will likely be replaced by the next calculation before the robot ever gets to them. The researchers built a training system that only looks at the second zone—the actions the robot actually executed. They teach the robot to adjust its behavior based solely on the consequences of the movements it really made, effectively cutting off the learning signal from the parts of the plan that were discarded. This ensures the robot learns from reality, not from a hypothetical future that never happened.

To test this, the researchers set up three challenging physical tasks on a mobile robot with two arms. The first task involved tossing an object into a bin, a motion that requires a continuous, unbroken swing. If the robot hesitates or jerks at the moment it switches between calculation cycles, the throw fails. The second task required the robot to cap a pen with extreme precision, aligning two small objects within a tolerance of just a few millimeters. The third task involved slicing open a cardboard box by guiding a blade along a tiny seam, a job that demands millimeter-level accuracy to avoid cutting the box itself. In all three cases, the robot started with a pre-trained brain that was good but not perfect. When the researchers let the robot practice using their new asynchronous learning method, the results were dramatic. The robot's ability to toss the object successfully jumped from thirty-nine percent to ninety-four percent. Its skill at capping the pen rose from a mere eight percent to eighty-three percent, and its success rate at opening the box improved from thirty percent to ninety percent.

The improvement was not just about getting the task done more often; it was about how the robot moved. The researchers measured the smoothness of the robot's motion and found that the new method reduced sudden jolts and jerky accelerations by nearly half. This smoothness was critical for the dynamic tossing task, where a pause in motion would cause the object to fall short. The system also proved flexible enough to handle human help. If a human operator needed to step in to correct a mistake, the robot could absorb that human action directly into its learning process without needing to translate it into a different code. The human's correction became just another example of the right way to move, allowing the robot to learn from both its own trials and human guidance simultaneously.

The researchers found that the robot's learning was not always a straight line. In the box-opening task, the robot's performance actually dipped before it improved, suggesting that the system was exploring new ways to move that initially seemed worse before finding the correct path. This indicates that the robot was truly learning to adapt rather than just memorizing a fixed set of movements. However, the researchers also noted the limits of their approach. The robot's ability to learn is still tied to the quality of its initial pre-trained brain. If the base robot is fundamentally confused about a task, the small adjustments made by this learning system may not be enough to fix the problem. Furthermore, the system relies on the robot's calculations finishing within a strict time limit; if the computer gets too slow, the timing of the movements can fall out of sync, destabilizing the whole process.

Ultimately, this work demonstrates that for robots to become truly useful in the real world, their learning algorithms must match the messy, overlapping reality of how they move. By teaching the robot to focus only on the actions it actually completed and to ignore the ones it discarded, the researchers have created a path for machines to learn complex, high-speed tasks without sacrificing the smoothness required to perform them. The result is a robot that can throw, cap, and cut with a level of reliability and fluidity that was previously out of reach, proving that the key to better robot learning lies in understanding the precise moment when a plan becomes reality.

Technical Summary: SmoothRL

1. Problem Statement

Deploying state-of-the-art generalist robot policies (e.g., Vision-Language-Action models and World-Action Models) in the physical world faces a fundamental conflict between reliability and smooth real-time execution.

Reliability: Achieving robustness for diverse real-world tasks (e.g., cutting sealed packages) requires sample-efficient online Reinforcement Learning (RL) to adapt pretrained models.
Smooth Execution: The increasing scale of foundation models leads to high inference latency. Synchronous execution (waiting for inference before acting) introduces pauses at chunk boundaries, degrading performance in dynamic tasks.
The Mismatch: Modern systems resolve latency by using asynchronous inference (overlapping policy computation with action execution) and action chunking. However, existing online RL methods typically assume synchronous execution. This creates a distributional shift: policies are optimized under dynamics (synchronous) that differ from those encountered at deployment (asynchronous). Specifically, in asynchronous execution, a generated action chunk is only partially executed before being superseded by a new chunk. Standard value-gradient updates would incorrectly backpropagate through "discarded" actions that never reached the environment, contaminating the learning signal.

2. Methodology: SmoothRL

SmoothRL is an online RL framework designed to fine-tune pretrained policies within an asynchronous inference loop. It adheres to three core requirements: Online learning, Asynchronous execution, and Value-gradient optimization.

2.1. Asynchronous Inference Loop and Chunk Partitioning

To resolve the mismatch between generated and executed actions, SmoothRL explicitly models the asynchronous process during training. Under a fixed latency budget $n$ (where inference time is capped), every generated action chunk of horizon $H$ is partitioned into three distinct regions based on frame index:

Committed Region $[0, n)$ : Actions already issued by the previous inference cycle. These are fixed and cannot be altered by the current policy.
Execution Region $[n, 2n)$ : The newly generated actions that are actually executed by the robot before the next chunk arrives.
Discarded Region $[2n, H)$ : Actions generated in the current chunk that are superseded by the next inference cycle before they can be executed.

2.2. Gradient Truncation and Value-Gradient Paradigm

SmoothRL follows the value-gradient paradigm, where the action-value function $Q$ updates policy parameters $\theta$ via $\nabla_\theta J = \mathbb{E}[\nabla_a Q(s, a) \cdot \nabla_\theta \pi_\theta(s)]$ . To ensure correctness under asynchrony:

Gradient Truncation: The value gradient $\nabla_a Q$ is computed only with respect to the Execution Region ( $a[n, 2n)$ ). Gradients are not propagated through the Committed or Discarded regions. This ensures policy updates depend exclusively on the consequences of actions actually executed.
Critic Conditioning: The critic retains the Committed Region as input. This is necessary for two reasons:
1. Unbiased Bootstrap: The critic must be conditioned on the full action sequence that generated the interval reward to ensure the chunk-skip bootstrap is unbiased.
2. State Augmentation: The committed region represents the "in-flight" actions, providing the necessary state augmentation to model the concurrent decision process (where the next chunk is generated while the current one is executing).

2.3. Training Loop and Human Intervention

Embedded Asynchrony: The asynchronous inference loop runs during training rollouts, not just at deployment. Replay trajectories record the actions the robot actually executed under the same timing schedule, ensuring the policy is never optimized under unseen dynamics.
Raw Action Space: The framework operates in raw action space rather than latent space. This allows human interventions (teleoperation) to be incorporated directly as regression targets without requiring latent inversion.
Intervention Modes: Two modes are supported:
- Absolute Intervention: The teleoperated chunk replaces the policy output entirely.
- Residual Intervention: The human input adds a correction to the policy output.
  Both modes are treated identically downstream: the actual issued action is recorded in the committed region and used as the Behavioral Cloning (BC) target.

2.4. Instantiation

The authors implement SmoothRL using a frozen base policy (e.g., $\pi_{0.5}$ ) and a lightweight, trainable attached network (Actor-Critic).

The base policy generates a reference action chunk and a latent token.
The attached network predicts a bounded residual correction to the reference action.
The Actor loss includes a $Q$ -maximization term, a BC anchor (regularizing against the reference action), and a smoothness regularizer enforcing velocity, acceleration, and jerk constraints to ensure continuity across chunk boundaries.

3. Key Contributions

Framework for Asynchronous Online RL: SmoothRL is the first framework to integrate online RL fine-tuning directly into an asynchronous inference loop, resolving the "Reinforce in Deployment" mismatch.
Chunk-Role Partitioning: The explicit partitioning of action chunks into Committed, Execution, and Discarded regions allows for precise gradient management.
Gradient Truncation: The method introduces a mechanism to truncate value gradients to the Execution Region, ensuring that policy updates are driven solely by executed actions while the critic remains conditioned on the full history for unbiased bootstrapping.
Direct Human Integration: By operating in raw action space, the framework naturally incorporates human interventions as both supervised targets and value-labeled transitions, facilitating safer and more efficient real-robot learning.

4. Experimental Results

The framework was evaluated on the Astribot S1 mobile bimanual robot across three tasks requiring high precision and dynamic control:

Dynamic Tossing: Throwing an object into a target bin (requires continuous motion and precise release timing).
Pen Capping: Aligning a pen with a cap and securing it (requires millimeter-level relative pose accuracy).
Box Opening: Slitting a tape seam with a blade (requires sub-millimeter insertion precision).

Performance Improvements:
After 250 online rollout episodes, SmoothRL significantly outperformed the frozen base policy:

Dynamic Tossing: Improved from 39% to 94% success rate.
Pen Capping: Improved from 8% to 83% success rate.
Box Opening: Improved from 30% to 90% success rate.

Analysis:

The base policy exhibited systematic biases (e.g., leftward bias in box opening, velocity modulation errors in tossing).
SmoothRL effectively corrected these systematic offsets.
In "Box Opening," performance initially dipped (to 20%) before recovering, indicating the exploration required to discover effective corrections.
The framework reduced motion jerk and acceleration by 47% and 52% respectively compared to the base policy, demonstrating improved smoothness.

5. Significance and Limitations

Significance:
The paper argues that for real-world deployment, online RL must be performed under the same asynchronous execution paradigm used during deployment. SmoothRL bridges the gap between the high-latency requirements of large foundation models and the need for sample-efficient, real-time adaptation. It demonstrates that by explicitly modeling the asynchronous schedule and truncating gradients to executed actions, one can achieve high-precision, dynamic control on real robots without sacrificing the benefits of large pretrained models.

Limitations (as stated by the authors):

Inference Latency Constraint: The framework relies on a fixed latency budget. If inference latency fluctuates beyond this budget, the scheduled chunk handover fails, destabilizing the loop.
Expressiveness Bounded by Base Policy: The RL module's capacity is constrained by the frozen base policy's internal representations and the bounded nature of the residual corrections. If the base policy exhibits large systematic biases outside the local neighborhood of the reference action, the current instantiation may lack the expressiveness to fully rectify them.

Future Directions:
The authors suggest extending the framework to full generative policy updates (not just residuals), investigating the interaction between policy learning and action blending, and scaling evaluation to broader task distributions to better understand the limits of local correction mechanisms.