🤖 machine learning

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

The paper introduces PARTS, a real-world reinforcement learning framework that fine-tunes pretrained robot policies by focusing human intervention only on critical subtask bottlenecks, thereby significantly improving long-horizon manipulation success rates with minimal human effort compared to existing methods.

Original authors: Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun

Published 2026-09-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled with tasks that require a sequence of precise steps, like opening a box, picking up an object, and placing it inside. For years, engineers tried to teach robots by showing them thousands of examples of how to do a whole job, hoping the machine would learn the entire routine at once. More recently, a new generation of robot "brains" has emerged. These are trained on massive collections of video and data, allowing them to understand language and perform many different actions without specific training for each one. They are surprisingly good at the basics: they can pick up a cup, move it across a table, or open a door. However, when faced with a long, complicated task that demands high precision, these general-purpose robots often stumble at a few specific, difficult moments. They might successfully grab an object but hold it in the wrong way, causing the next step to fail. The challenge for scientists is how to fix these specific weak points without wasting time re-teaching the robot everything it already knows how to do well.

A team of researchers has developed a new method called PARTS to solve this problem. Instead of trying to retrain the robot on the entire task from start to finish, their approach focuses exclusively on the moments where the robot fails. Imagine a robot that can open a charging case and hold it steady, but repeatedly fails to slide an earbud into the tiny slot. The researchers identified this specific failure point as a "bottleneck." Rather than having the robot practice the whole routine over and over, they froze the robot's general knowledge and trained a small, specialized add-on only for that one difficult step. This add-on learns to make tiny, precise corrections to the robot's movements just when they are needed. The system uses a computer program to watch the robot, decide when it is entering a difficult phase, and switch in the specialized helper. Once the difficult step is done, control returns to the robot's original, reliable brain. This cycle allows the robot to practice the hard part repeatedly without needing a human to reset the scene or correct its mistakes every time.

The researchers tested this method on real robots performing complex tasks, such as inserting earbuds into a case, sorting small LEGO bricks, and plugging a cable into a router. In one experiment with a two-armed robot, the original version could only complete the entire earbud task about 32% of the time. After using their targeted training method, the success rate jumped to 61%. In another test with a single-arm robot plugging in a cable, the success rate soared from 50% to 95%. These improvements were achieved in just tens of minutes of real-world practice per task. The team compared their method against other ways of training robots, where the entire task is practiced from the beginning. Those other methods required the same amount of robot time but resulted in much lower success rates, often failing to improve the robot's performance at all. The researchers found that by concentrating learning only on the specific steps where the robot struggled, they could achieve far better results with less effort and less human supervision.

The key to this success lies in how the system handles failure. In traditional training, if a robot fails at the very last step of a long task, it receives no credit for the many steps it got right, making it hard to learn. The new system breaks the task down and rewards the robot immediately after it successfully completes just the difficult sub-step. If the robot manages to insert the earbud, it gets a positive signal right away, even if the case wasn't closed perfectly yet. This frequent feedback helps the robot learn faster. Furthermore, the system includes a smart way to organize the practice data. When the robot finally succeeds at a difficult step, that successful attempt is saved and used to retrain the specialized helper more thoroughly, ensuring it doesn't forget what worked. This process happens automatically, with the robot resetting itself or asking for a human reset only when absolutely necessary, such as when an object falls to the floor.

The study demonstrates that robots do not need to be retrained from scratch to become better at complex jobs. By identifying the few specific moments where they struggle and applying focused, targeted practice to those moments, engineers can significantly boost a robot's ability to complete long, difficult tasks. This approach respects the robot's existing capabilities, keeping the parts that work well exactly as they are while strengthening the weak links. The results suggest a practical path forward for deploying robots in real-world settings, where tasks are often long and require a mix of general skill and precise execution. The researchers conclude that this method of concentrating effort on bottlenecks allows robots to learn more efficiently, requiring less human intervention and achieving higher reliability than previous methods.

Technical Summary: PARTS (Policy Adaptation with RL on Targeted Subtasks)

Problem Statement

Pretrained robot foundation policies, particularly Vision-Language-Action (VLA) models, often demonstrate strong zero-shot capabilities across diverse manipulation tasks. However, when applied to long-horizon tasks, these policies frequently fail at specific "bottleneck" subtasks (e.g., high-precision insertion or handling out-of-distribution objects) while successfully executing the remainder of the sequence.

Existing approaches to bridge this gap face significant limitations:

Supervised Fine-Tuning (SFT): Collecting full-task demonstrations for SFT is inefficient because it requires humans to repeat behaviors the policy already performs well, focusing effort only on the few failure points.
Full-Task Reinforcement Learning (RL): Applying RL to the entire long-horizon task using sparse terminal rewards (success only if the entire task succeeds) is data-inefficient. Early failures in the sequence prevent the agent from reaching later subtasks to learn, and the sparse reward signal provides no feedback for intermediate improvements.
Human Intervention: Many real-world RL methods require continuous human monitoring, corrective teleoperation, or manual policy switching during rollouts, limiting scalability.

The core challenge is to adapt a pretrained policy to a specific long-horizon task by correcting only the critical bottlenecks while retaining the reliability of the pretrained model for the rest of the task, all with minimal human intervention during the training phase.

Methodology: PARTS Framework

The authors propose PARTS (Policy Adaptation with RL on Targeted Subtasks), a framework that isolates bottleneck subtasks for targeted RL training while keeping the base policy frozen.

1. Architecture and Policy Structure

Frozen Base Policy: A pretrained VLA (specifically $\pi_{0.5}$ in this work) is fine-tuned via SFT on a small offline dataset to create a task-specific base policy ( $\pi_0$ ). This policy executes the entire task sequence nominally.
Residual Policies: For each identified bottleneck $k$ , a lightweight residual actor-critic network is trained. This network outputs a correction vector $U_{k,t}$ added to the base policy's nominal action chunk $\bar{A}_t$ .
Action Composition: The final command is a masked sum of the nominal action and the residual correction, restricted to specific action dimensions (e.g., gripper openness or end-effector translation) relevant to the bottleneck.

2. Agentic Scaffolding (Automation)

To enable autonomous training rollouts, PARTS utilizes "coding agents" (LLMs) to generate executable programs based on human-defined contracts. These programs manage the execution loop without human intervention:

Policy Selector: Monitors the state (via VLMs and geometric predicates) to determine when a specific bottleneck's entry conditions are met, activating the corresponding residual policy.
Success Verifier: Evaluates the outcome of a subtask attempt. It uses visual evidence (e.g., object masks, state queries) to provide a sparse local reward ( $r_k \in \{0, 1\}$ ) immediately upon subtask completion, rather than waiting for full-task success.
Auto-Reset Policy: Decides whether to retry a subtask from the current state or reset the scene. If the robot can autonomously restore the scene (e.g., dropping an object), it does so; otherwise, it requests a human reset.

3. Learning Algorithm

Online RL: The system uses TD3+BC (Twin Delayed DDPG with Behavior Cloning) for each bottleneck. The critic estimates the value of residual corrections, while the actor is regularized to stay close to successful corrections (imitation) and anchored to the nominal policy for failed attempts.
Success-Reweighted Retraining: To address the sparsity of rewards and overfitting to specific reset configurations, the system periodically retrains the residual policy on a curated dataset. This dataset includes all successful episodes and a sampled fraction ( $\rho$ ) of failed episodes. This process emphasizes successful local experiences and redeploys the updated policy to collect new data, creating a feedback loop between policy improvement and data collection.

Key Contributions

Formulation of Targeted Subtask RL: A novel approach to adapting pretrained policies by isolating and training only on bottleneck subtasks using local outcome supervision, while evaluating reliability over the complete long-horizon execution.
The PARTS Framework: A system integrating executable subtask supervision (selectors/verifiers), residual RL, and success-reweighted retraining. This enables training rollouts that require no human corrections, policy switching decisions, or manual reward labeling during execution.
Empirical Validation: Experiments demonstrating that PARTS significantly boosts full-task success rates on bimanual and single-arm platforms under limited real-world rollout budgets, outperforming existing full-task RL fine-tuning baselines.

Experimental Results

The framework was evaluated on three long-horizon tasks across two real-robot platforms:

Earbud Insertion (Bimanual YAM): A dependent chain requiring high precision.
- Result: Full-task success increased from 32% to 61%.
LEGO Sorting (Bimanual YAM): Sorting 10 bricks (out-of-distribution for the base policy) into bins.
- Result: Progress score (fraction of bricks sorted) increased from 54% to 82%.
Cable Unplug & Plug (Single-arm Franka FR3): Requiring precise alignment.
- Result: Full-task success increased from 50% to 95%.

Comparison with Baselines:
Under a matched robot-rollout budget, PARTS outperformed:

Frozen Base Policy: Significant gains across all tasks.
DSRL: A method learning in the latent noise space of the VLA.
EXPO-FT: A method editing VLA action chunks while updating the backbone.
RLT: A method using a single VLA-to-RL handoff for a critical phase.

PARTS achieved higher success rates (often >25% improvement) because it focused learning on bottlenecks and provided frequent local rewards, whereas baselines wasted rollout time on reliable subtasks or struggled with sparse terminal rewards.

Ablation Study:
Removing success-reweighted retraining resulted in lower performance, confirming that periodically retraining fresh networks on curated successful data helps escape local optima and improves generalization to the evaluation distribution.

Significance and Claims

The paper claims that PARTS represents a shift from learning behaviors from scratch to targeted refinement of capable pretrained policies. By concentrating real-world interaction only where the policy fails, the framework:

Reduces Human Involvement: Eliminates the need for corrective teleoperation or manual switching during training, requiring human input only for initial bottleneck identification and physical resets when the robot cannot self-correct.
Improves Data Efficiency: Achieves high full-task success with only tens of minutes of real-world RL rollouts per task.
Preserves Reliability: Maintains the robust behaviors of the pretrained model for non-bottleneck subtasks, avoiding the degradation of reliable behaviors often seen in full-task fine-tuning.

The authors conclude that this approach offers a scalable path for adapting generalist robot policies to specific, complex long-horizon tasks in the real world. Future work envisions further automation where agents identify bottlenecks and generate dense rewards autonomously.