💻 computer science

Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations

The paper proposes DR-LfD, a framework that integrates visuomotor policies with task and motion planning (TAMP) by decomposing demonstrations into atomic skills, thereby enabling robust, data-efficient long-horizon robotic manipulation that overcomes the brittleness of traditional planning and the generalization limits of pure imitation learning.

Original authors: Yizhou Chen, Hang Xu, Dongjie Yu, Yupu Lu, Tengye Xu, Zeqing Zhang, Wei Zhang, Yi Ren, Ben M. Chen, Jia Pan

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yizhou Chen, Hang Xu, Dongjie Yu, Yupu Lu, Tengye Xu, Zeqing Zhang, Wei Zhang, Yi Ren, Ben M. Chen, Jia Pan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot to do a complex magic trick, like juggling while walking a tightrope. For a long time, scientists have tried two main ways to teach robots. The first way is like giving the robot a strict, step-by-step recipe written by a human. This is great for logic, but if the robot drops a spoon or the table moves slightly, the recipe breaks, and the robot freezes. The second way is like showing the robot a video of a human doing the trick and hoping the robot learns by watching. This is great for copying movements, but if the robot sees the spoon in a slightly different spot than it did in the video, it gets confused and fails. The big question in robotics is: how do we get the best of both worlds? How do we make a robot that is smart enough to plan ahead but flexible enough to handle real-world messiness without needing to watch millions of videos to learn every single possibility?

This paper introduces a new system called DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations) that tries to solve this puzzle. Think of it as a master chef who doesn't just memorize one giant recipe for a seven-course meal. Instead, the chef breaks the meal down into tiny, perfect "skills" like chopping an onion, searing a steak, or whisking eggs. The chef practices each of these small skills a few times until they are perfect. Then, when a customer orders a new, strange dish, the chef doesn't panic. They simply look at the order, pick the right pre-practiced skills from their mental list, and rearrange them in a new order to cook the new meal.

The paper suggests that by breaking long, complicated tasks into these small, reusable "skills," a robot can learn much faster and handle new situations much better. Instead of needing to learn every possible combination of a 20-step task (which would take forever), the robot only needs to learn the 20 individual steps once. Then, a smart "planner" (the chef's brain) figures out how to put those steps together for any new job.

Here is how DR-LfD works, using the analogy of a robot learning to play a complex video game:

1. Breaking the Game into Levels (Decomposition)
When a human shows the robot how to do a task (like packing a screwdriver or handing a cup to a friend), the system doesn't just record the whole thing as one long video. It uses a special "contact detector" to watch for when the robot's hand touches an object or lets go. It slices the video into tiny chunks based on these touches.

The "Simple Moves": For easy parts, like picking up a cup or placing it down, the robot learns a "primitive." This is like a pre-programmed muscle memory that knows exactly how to move the arm to grab an object, no matter where it is on the table.
The "Hard Moves": For tricky parts, like wiping a cup or passing an object between two hands, the robot learns a "visuomotor policy." This is like a reflex that watches the camera and adjusts its hands in real-time to keep the object steady.
The "Bridge Moves": For moving through empty space, the robot just uses standard math to plan a path.

2. The Smart Planner (Reorganization)
Once the robot has a "toolbox" of these skills, it doesn't just run them one after another blindly. It uses a Task and Motion Planner (TAMP). Think of this planner as a GPS for the robot's brain.

If the robot needs to hand a cup to a person, the planner checks: "Is the cup reachable? Is the person's hand in the right spot? Is there a chair blocking the way?"
If the cup is too far away, the planner doesn't just say "fail." It says, "Okay, let's first move the chair, then pick up the cup, then hand it over."
If the robot tries to grab a cup and misses because the cup moved, the planner notices, stops the action, and recalculates a new path. It's like a GPS rerouting you when there's traffic.

3. Testing the System
The authors tested this system in both computer simulations and with real robots (using a dual-arm robot called ALOHA). They gave the robot a small amount of training data—just 20 demonstrations for each skill.

The Results: When they tested the robot with objects in new, weird places (places it had never seen before), the old methods (like just watching videos) failed often. But DR-LfD kept succeeding. For example, in a "peg-in-hole" task, while other methods failed about half the time when the hole was moved, DR-LfD succeeded 100% of the time in standard tests and 88% in very difficult, random tests.
Handling Obstacles: In one test, they put a pole in the way that blocked the robot's arm. The robot realized it couldn't reach the target, so the planner told it to move the pole first, then reach for the target. It successfully cleared the obstacle and finished the job.
Long Tasks: They even made the robot do long, multi-step tasks, like handing off three different tapes to a basket, or packing a screwdriver while wiping a cup. In these specific experiments, the robot successfully chained together 19 to 21 different steps, a feat that usually causes robots to get confused and fail. However, the paper notes that this success is within the computational limits of the planner and specific to the tasks they tested.

What the Paper Says It Can't Do (Yet)
The authors are careful to note that this isn't magic. The system still needs humans to specify the goals or provide preferences; it cannot autonomously "figure out what needs to be done" without that input. Also, because the robot relies on 3D depth cameras to see objects, if the camera is dirty or the lighting is bad, the robot might get confused. The paper also mentions that if the task gets too complicated with too many objects, the planning part takes longer to think, just like a human getting overwhelmed by too many choices.

The Bottom Line
The paper suggests that by teaching robots to learn small, reusable skills and then using a smart planner to mix and match them, we can build robots that are much more flexible and require far less training data than before. It's a step toward robots that can walk into a messy room, figure out what needs to be done (once a human tells them the goal), and do it without needing a million practice videos for every single possible scenario.

Technical Summary: DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations)

1. Problem Statement

Long-horizon robotic manipulation requires frameworks that can simultaneously perform high-level reasoning (sequencing skills) and fine-grained execution (contact-rich operations). Existing approaches face a fundamental trade-off:

Task and Motion Planning (TAMP): Excels at symbolic reasoning and constraint satisfaction but often relies on manually engineered models and precise perception, making it brittle in contact-rich, unstructured environments.
Imitation Learning (IL): Effective at replicating complex visuomotor behaviors from data but suffers from poor spatial generalization and an inability to handle multi-stage transitions. End-to-end policies must implicitly learn the combinatorial logic of when and how to switch skills, requiring exponentially large datasets to generalize to unseen setups or long sequences.

The core challenge is the combinatorial complexity of long-horizon tasks. Learning a policy for a sequence of $M$ skills from $N$ available primitives typically demands data scaling exponentially with $M$ . Furthermore, current hybrid systems often struggle with defining robust skill-switching criteria and handling physical constraints (reachability, collision) when integrating learned policies.

2. Methodology: DR-LfD Framework

DR-LfD proposes a systematic framework that decomposes human demonstrations into atomic skills and reorganizes them via a TAMP-gated decision-making system. The framework operates in two phases: Offline Skill Acquisition and Online Plan Composition.

A. Skill Decomposition and Abstraction

The system processes demonstration data to extract atomic skills based on contact relationships:

VLM-Assisted Grounding: A Vision-Language Model (VLM, specifically Qwen-VL-Max) analyzes video demonstrations to identify task objects, their properties (e.g., container, surface), and a sequence of discrete skills. It grounds edge operations (gripper-object or object-object interactions) to specific skills.
Contact-Aware Partitioning: Demonstrations are segmented into distinct phases (initiation, contact-rich, termination) by detecting changes in contact states (e.g., distance thresholds between grippers and objects).
Graph Representation: Each skill is abstracted into a precondition graph ( $G_{pre}$ ) and an effect graph ( $G_{eff}$ ). These graphs represent the contact signatures (e.g., "gripper holds object," "object on table") using a scene-graph structure, abstracting away stochastic intermediate behaviors.

B. Skill Learning and Categorization

Skills are categorized and learned using different representations to maximize data efficiency and generalization:

Object-Centric Primitives (Low Complexity): For simple skills (e.g., pick, place, drop), the system learns open-loop trajectories using an SO(3)-equivariant diffusion model. This network takes point clouds and language descriptions as input to predict end-effector trajectories, finger joint angles, and contact states. The equivariance ensures robust spatial generalization to novel object placements without requiring massive data augmentation.
Visuomotor Policies (Contact-Rich): For complex, contact-rich skills (e.g., wiping, handoff), the system learns a closed-loop visuomotor policy (e.g., using Diffusion Policy). Crucially, a KeyPose Predictor (a diffusion-based MLP) is trained to predict the initial and terminal proprioceptive states (keyposes) required to initiate and terminate these policies. This ensures the policy starts in an in-distribution state.
Planned Trajectories: Skills involving no contact (e.g., transit) are handled by classical motion planners.

C. Integration with TAMP (Stream Functions)

Learned skills are encoded as stream functions compatible with the PDDLStream framework:

Stream Functions: These act as conditional generators that map symbolic preconditions to continuous variables (poses, trajectories) or binary safety checks.
Automated Schema Construction: An algorithm automatically translates the contact graphs ( $G_{pre}, G_{eff}$ ) and edge operations into PDDL predicates and associates them with the appropriate stream functions (e.g., LearnedAttach, LearnedUniKeyPose).
Constraint Enforcement: The framework implements specific test streams for physical constraints:
- Reachability: A precomputed map checks if an object is reachable by a specific arm configuration.
- Policy Safety: A Minimum Distance Field (MDF) is constructed from demonstration trajectories to define "safe zones." The planner verifies that unexpected obstacles do not violate these safety constraints before executing a policy.

D. Online Execution and Adaptation

The online phase employs a perceive-plan-act-verify loop:

Global Skeleton: A coarse TAMP solver generates a high-level skill sequence (skeleton) based on the current state and goal.
Local Replanning: For each skill in the skeleton, the system refreshes the state via perception, solves a local TAMP problem to generate detailed motions (or keyposes for policies), and executes the command.
Verification: A contact detector monitors the execution. If a skill fails (e.g., missed grasp) or the state drifts, the system triggers local replanning for the same subgoal rather than failing the entire task.

3. Key Contributions

TAMP-Gated Decomposition: A novel system that abstracts atomic skills from demonstrations and reorganizes them via TAMP to achieve combinatorial goals, shifting the data burden from exponential sequence length to the number of distinct skill types.
Equivariant Primitives: The use of equivariant diffusion models to learn object-centric primitives, enabling robust spatial generalization from minimal data (e.g., 20 demonstrations per skill).
TAMP-Compatible Policy Modeling: A systematic method for modeling the initiation, termination, and constraints of learned visuomotor policies (via KeyPose predictors and MDF safety checks) in a form compatible with sampling-based TAMP.
Comprehensive Validation: Demonstration of the framework's ability to handle long-horizon, multi-step tasks with unseen setups and physical constraints in both simulation and real-world dual-arm (ALOHA) hardware.

4. Experimental Results

The authors evaluated DR-LfD across simulated and real-world benchmarks, comparing it against state-of-the-art imitation learning (IL) methods (ACT, DP, $\pi_0.5$ ) and TAMP baselines.

Spatial Generalization (OOD): In the "Peg-in-Hole" task, DR-LfD achieved 100% success in in-distribution (ID) and 88% in highly out-of-distribution (XYH-OOD) settings, significantly outperforming baselines (e.g., DP achieved 12% in XYH-OOD).
Real-World Dual-Arm Tasks: On tasks like Object Handoff, Screwdriver Packing, and Cup-sleeve Insertion, DR-LfD consistently outperformed DP and matched or exceeded the pre-trained $\pi_0.5$ model, particularly in OOD and "Leaky" (complex displacement) scenarios.
Constraint Handling: In "Unreachable" and "Unsafe" scenarios, DR-LfD successfully replanned to move obstacles or reposition objects to satisfy reachability and collision constraints, achieving 55% and 70% success rates respectively, whereas baselines failed entirely.
Long-Horizon Composition: The system successfully composed skills into 19-21 step plans (e.g., "Three-Tapes" and "Cup-Sponge-Screwdriver" tasks) using a small repertoire of learned skills.
Data Efficiency: On the DexMimicGen dataset, DR-LfD trained on 100 demonstrations outperformed Diffusion Policy trained on 1,000 demonstrations, highlighting the efficiency of the decomposition approach.

5. Significance and Claims

The paper claims that DR-LfD addresses the fundamental limitation of end-to-end learning by decoupling skill execution from skill sequencing. By treating learned skills as modular, TAMP-compatible components, the framework:

Reduces Data Requirements: It avoids the need for exponential datasets by learning skills in isolation and recombining them.
Enhances Reliability: It incorporates physical constraints (reachability, collision) directly into the planning loop, allowing the system to recover from failures and handle unseen object placements.
Bridges the Gap: It effectively integrates the flexibility of visuomotor policies with the reasoning capabilities of symbolic planning, making it suitable for complex, contact-rich, long-horizon manipulation tasks in real-world settings.

The authors acknowledge limitations inherent to TAMP (exponential complexity with increasing objects) and equivariant learning (dependence on depth sensing), suggesting future work in automated goal specification and depth-free equivariant learning. However, they assert that the current framework provides a robust, data-efficient pathway for deploying dexterous robots in unstructured environments.