💻 computer science

Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning

This paper proposes an agentic reinforcement learning framework that enhances robotic manipulation robustness by introducing runtime execution quality metrics and a high-level decision-making policy that detects degradation and triggers recovery mechanisms to restore nominal task states, achieving significant success rate improvements on the LIBERO benchmark.

Original authors: Xiaopeng Zhang, Yueyang Weng, Qi Liu, Yongjin Mu, Yanjie Li

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaopeng Zhang, Yueyang Weng, Qi Liu, Yongjin Mu, Yanjie Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to build a complex Lego castle. You give it a set of instructions, and it starts stacking bricks. But robots are clumsy; sometimes they miss a brick by a millimeter, or the table shakes, or their "grip" slips just a tiny bit. In the world of robotics, this is called uncertainty. When a robot makes a small mistake early on, that error doesn't just disappear; it gets worse and worse as the task goes on, like a snowball rolling down a hill. This is known as compounding error. Eventually, the robot might be holding a brick in the wrong spot, or it might have dropped a piece entirely, but it keeps trying to follow its original plan anyway, leading to a total mess.

For a long time, scientists tried to fix this by teaching the robot more examples of every possible mistake it could make. But that's like trying to memorize every single way a Lego tower can fall over before you even start building—it takes forever and costs a fortune. Another idea was to give the robot a super-smart "brain" (like a language model) to constantly check its work and tell it what to do next. But that makes the robot slow and complicated, like having a teacher shouting instructions at every single brick placement. The big question is: Can we make a robot that notices when it's going off-track and knows how to fix itself, without needing a supercomputer or a library of every possible disaster?

This paper introduces a clever new way to handle robot mistakes, called Agentic Reinforcement Learning. Instead of trying to teach the robot how to move its arms perfectly (which is the hard part), the authors built a "manager" that watches the robot's performance and decides when to keep going, when to back up, and when to start over. Think of the robot's arm as a skilled but slightly clumsy worker, and this new "manager" as a foreman who isn't doing the lifting but is watching the whole construction site.

The researchers found that when the robot starts to stumble, this manager can spot the trouble before it becomes a disaster. They created two "scorecards" to measure how well the robot is doing: one checks if the robot is moving smoothly right now (local quality), and the other checks if the robot is still on the right path compared to a perfect example (global quality). If the scores drop, the manager doesn't try to invent a new way to move the arm. Instead, it chooses from a small, pre-defined list of recovery moves:

RETRY: If the robot just missed a grip, the manager tells it to back up a few steps and try again.
REPAIR: If the robot is stuck or holding something weirdly, the manager tells it to let go, move to a safe spot, and reset its grip.
RESET: If the robot has dropped everything or is in a hopeless position, the manager hits the "restart" button and begins the task from the very beginning.

The team tested this system on a famous set of robot challenges called LIBERO, which involves tasks like picking up bowls, opening drawers, and stacking objects. They found that adding this "manager" made the robots much better at finishing their jobs, even when things went wrong. In standard tests, the success rate went up by as much as 13.7%. But the real magic happened when they added random disturbances, like shaking the robot or messing up its movements. In those chaotic situations, the success rate jumped by up to 39.2%.

The paper argues that simply training robots on more data isn't always the best answer, and adding heavy, slow reasoning systems isn't efficient either. Instead, this "agentic" approach separates the decision to recover from the action of moving. The robot doesn't need to learn new skills; it just needs a smart supervisor to know when to say, "Wait, that didn't work, let's try that again," or "Okay, we're stuck, let's start over." The results show that this method works well with different types of robot brains, from simple ones to complex ones, making robots more robust and reliable without needing to retrain them from scratch. While the system can't fix every impossible situation, it proves that a little bit of smart oversight can go a long way in keeping a robot on its feet.

Technical Summary: Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning

1. Problem Statement

Robotic manipulation faces fundamental challenges due to uncertainty, contact-rich interactions, and long-horizon execution. While recent Vision-Language-Action (VLA) models demonstrate strong generalization through large-scale pretraining, they typically lack explicit mechanisms to assess execution stability or recover when the robot deviates from its nominal trajectory. Small errors in perception or dynamics can accumulate over time, causing "execution degradation." This leads to unstable behaviors, loss of task progress, or irreversible failures (termed "execution-level failures").

Existing mitigation strategies have limitations:

Data Scaling: Expanding training data and fine-tuning VLAs is resource-intensive and often ineffective against long-tail failure cases.
External Reasoning: Introducing Vision-Language Models (VLMs) for failure detection and replanning adds system complexity, inference latency, and sensitivity to instruction distribution shifts.
Imitation Learning: Standard policies lack explicit mechanisms to adapt when execution deviates from expected behavior.

The core problem addressed is how to improve execution robustness without retraining the low-level policy or relying on heavy external reasoning modules.

2. Methodology

The authors propose an Agentic Reinforcement Learning (RL) framework that separates execution management from action generation. The system consists of a frozen low-level policy (e.g., a VLA) and a high-level agentic policy that regulates the execution process.

A. Execution Quality Metrics

To enable the agentic policy to make informed decisions, the authors introduce two complementary metrics to assess execution quality at runtime:

Local Execution Quality ( $q_{local}$ ): Monitors short-term stability by analyzing motion effectiveness (ratio of actual end-effector movement to commanded action magnitude) and motion smoothness (inverse coefficient of variation of speeds) within a sliding window.
Global Execution Quality ( $q_{global}$ ): Captures long-horizon drift by comparing the current execution prefix against a library of successful reference trajectories. It encodes recent execution windows into feature vectors and computes the Euclidean distance to the $k$ -nearest neighbors in a stage-aware reference library.
These metrics are aggregated into a single scalar score ( $q_{agg}$ ) to evaluate overall execution health.

B. Recovery Mechanisms

The agentic policy selects from a discrete set of execution modes $\mathcal{A} = \{ \text{EXECUTE, RETRY, REPAIR, RESET} \}$ :

EXECUTE: Continues the low-level policy's nominal execution.
RETRY: Rolls back the robot to the most recent state with the highest aggregated quality ( $q_{agg}$ ) and resumes execution. This handles mild degradations (e.g., slight grasp misalignment).
REPAIR: Performs a stronger rollback to a contact-free state with high $q_{agg}$ within a longer horizon. This handles obstructions or contact issues by disengaging and repositioning.
RESET: Terminates the current episode and restarts the task from the initial configuration for irrecoverable states (e.g., dropped objects).

These mechanisms utilize an operational space controller (OSC) to drive the robot back to historical states without generating new low-level actions.

C. Agentic Reinforcement Learning Formulation

The problem is formulated as a Partially Observable Markov Decision Process (POMDP):

Observation: A fixed-length sliding window of recent execution history, including proprioception, low-level actions, and execution quality scores.
Action: A categorical distribution over the four execution modes.
Reward Function: Designed to encourage task completion (+1.0) and penalize failure (-1.0), time steps (-0.02), and the invocation of recovery mechanisms (costs: -0.1 for RETRY, -0.3 for REPAIR, -0.5 for RESET).
Optimization: The agentic policy is trained using Proximal Policy Optimization (PPO). The actor uses only execution history, while the critic has access to privileged simulator states to improve value estimation.

3. Key Contributions

Runtime Execution Metrics: The proposal of two complementary metrics (local and global execution quality) to assess execution stability and drift at runtime without requiring privileged information.
Agentic Execution Framework: A novel formulation where a high-level policy acts as an "execution manager," selecting execution modes to regulate a frozen low-level policy. This separates execution regulation from action generation.
Lightweight Recovery: The design of specific, non-learning recovery mechanisms (RETRY, REPAIR, RESET) that restore the robot to previously visited nominal states, avoiding the need for retraining or complex semantic reasoning.
Modular Integration: The ability to apply this framework to various pre-trained policies (VLAs and Diffusion Policies) without modifying their weights.

4. Experimental Results

The method was evaluated on the LIBERO benchmark (covering Spatial, Object, Goal, and Long-horizon subsets) using baselines including OpenVLA, $\pi_0$ , $\pi_{0.5}$ , and Diffusion Policy.

Standard Conditions: The framework improved success rates across all baselines. The average improvement was +13.7% under standard settings, with the Diffusion Policy showing significant gains.
Disturbance Robustness: Under random kinematic disturbances, the method demonstrated substantial resilience. The average success rate improvement was +39.2%. For example, OpenVLA on the LIBERO-Long suite improved from 33.4% to 67.6% success rate under disturbance.
Decision Adaptivity: Analysis showed the agentic policy correctly adapts its choice of recovery mechanism based on degradation severity (e.g., selecting RETRY for mild noise, REPAIR for collisions, and RESET for irrecoverable states).
Efficiency: The overhead in episode length was modest (ranging from +5% to +15% depending on the baseline's fragility), indicating that recovery is triggered only when necessary.

5. Significance and Claims

The paper claims that this approach offers a substantially enhanced execution robustness by addressing execution-level failures directly. By decoupling execution management from action generation, the system evolves from a purely conditional action generator into a decision-driven framework capable of exploration and recovery.

The authors emphasize that this method:

Does not require retraining task-specific skills or foundation models.
Avoids the latency and complexity of dual-system architectures relying on VLMs.
Is highly amenable to sim-to-real deployment as it relies only on proprioceptive signals and low-level actions, not privileged simulator information.

Limitations: The authors modestly note that the method has limited ability to recover from extremely severe execution degradation or out-of-distribution scenarios where the low-level policy's nominal behavior is entirely invalid. Future work is planned to address these severe cases and validate the approach in real-world settings.