💻 computer science

DeCo: Task Decomposition and Skill Composition for Zero-Shot Generalization in Long-Horizon 3D Manipulation

The paper proposes DeCo, a model-agnostic framework that enhances zero-shot generalization in long-horizon 3D manipulation by decomposing demonstrations into reusable atomic skills and dynamically composing them via a vision-language model, achieving significant performance improvements across multiple imitation learning models and real-world tasks.

Original authors: Zixuan Chen, Junhui Yin, Yangtao Chen, Jing Huo, Pinzhuo Tian, Jieqi Shi, Yiwen Hou, Yinchuan Li, Yang Gao

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zixuan Chen, Junhui Yin, Yangtao Chen, Jing Huo, Pinzhuo Tian, Jieqi Shi, Yiwen Hou, Yinchuan Li, Yang Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🤖 The Big Problem: The "Robot Chef" Who Can't Read a Recipe

Imagine you have trained a robot chef. You've taught it three specific recipes:

Open the drawer.
Put a cookie in the open drawer.
Close the drawer.

The robot is a master at these three things. But then, you give it a new, slightly more complex instruction: "Put the cookie in the closed drawer, and then close it."

Even though the robot knows how to open, put things in, and close drawers, it freezes. It doesn't know how to break this new sentence down into the steps it already knows. It tries to do the whole thing at once, fails, and crashes.

This is the problem with current robots. They are great at following single instructions but terrible at combining those instructions to solve new, longer problems. They lack "common sense" composition.

💡 The Solution: DeCo (The "Lego Master")

The authors of this paper propose a framework called DeCo (Task Decomposition and Skill Composition). Think of DeCo as a Lego Master that sits between the human giving the order and the robot arm doing the work.

Here is how DeCo works, broken down into three simple steps:

1. Breaking Things Down (The "Atomic" Skills)

Instead of teaching the robot big, messy tasks, DeCo teaches it atomic skills—the smallest, indivisible Lego bricks of movement.

The Analogy: Imagine teaching a child to build a castle. Instead of saying "Build a castle," you teach them: "Place a block," "Stack a block," "Put a roof on."
How DeCo does it: It looks at old training videos and automatically cuts them into tiny clips based on when the robot's "hand" (gripper) opens or closes. It turns a 10-second video of "cleaning up" into three separate skills: "Pick up broom," "Sweep," and "Drop dust." Now the robot has a library of reusable Lego bricks.

2. The Smart Planner (The "Translator")

When you give the robot a new, long instruction (like "Clean the whole room"), DeCo uses a Vision-Language Model (VLM)—basically a super-smart AI that understands both pictures and words (like the brain behind ChatGPT or DALL-E).

The Analogy: Imagine you ask a human assistant, "I need to make a sandwich." The assistant doesn't just stare at you; they break it down: "Okay, first get the bread, then get the meat, then put it together."
How DeCo does it: The VLM looks at your new instruction, looks at the robot's "Lego library," and says: "Ah! To do this, the robot needs to: 1. Open the drawer, 2. Put the item in, 3. Close the drawer." It creates a step-by-step plan.

3. The Safety Bridge (The "Smooth Transition")

This is the secret sauce. Just because a robot knows how to "Open a drawer" and "Close a drawer" doesn't mean it can do them back-to-back without crashing. The end of one move might be in a weird position that makes the next move impossible.

The Analogy: Imagine a dancer. They know how to spin and how to jump. But if they spin and immediately jump without a smooth transition, they might trip. They need a "bridge" move to get from the spin to the jump gracefully.
How DeCo does it: It uses a Spatially-Aware Chaining Module. Before the robot tries to do the next step, DeCo calculates a safe, collision-free path to move the robot's hand from the "finish line" of the first skill to the "starting line" of the next skill. It ensures the robot doesn't bump into the table or the drawer while switching tasks.

🏆 The Results: From Zero to Hero

The researchers tested this on a new benchmark called DeCoBench (a video game-like test for robots).

Before DeCo: The robots were terrible at new, long tasks. If you asked them to do a 4-step or 10-step task they hadn't seen before, they failed 100% of the time. They were like a student who memorized the answers to three math problems but couldn't solve a fourth one that used the same math.
After DeCo: The same robots, using the same brain, suddenly became amazing.
- One robot's success rate jumped from 0% to 66%.
- Another jumped from 0% to 58%.
- In the real world (using a real robot arm), they went from failing every time to succeeding 53% of the time on completely new, long tasks, just by learning 6 basic skills first.

🚀 Why This Matters

This paper is a big deal because it doesn't require building a new, super-complex robot brain. Instead, it gives existing robots a new way of thinking:

Break big problems into small, known pieces.
Use a smart translator to figure out the order.
Build a safe bridge between the pieces.

It turns a robot that is a "parrot" (just repeating what it heard) into a robot that is a "problem solver" (capable of figuring out how to combine its skills to do something new). This is a massive step toward robots that can actually help us in our messy, unpredictable homes and workplaces.

1. Problem Statement

Current multi-task imitation learning (IL) models for robotic manipulation struggle with zero-shot generalization to novel, long-horizon tasks. While these models can learn specific skills (e.g., "open drawer," "place block"), they fail when presented with new instructions that require composing these known skills in unseen sequences (e.g., "put block in closed drawer and then close it").

The core challenges identified are:

Lack of Decomposition: Models cannot break down high-level, complex instructions into executable atomic steps.
Semantic-Physical Gap: Existing Vision-Language Model (VLM) planners often generate high-level plans that do not align with the low-level physical skill space, leading to execution failures.
Transition Instability: Sequentially executing skills often results in spatial discontinuities or collisions because the end pose of one skill does not naturally align with the start pose of the next.

2. Methodology: The DeCo Framework

The authors propose DeCo (Task Decomposition and Skill Composition), a model-agnostic framework that enhances existing multi-task IL models without requiring retraining on the new long-horizon tasks. It consists of three key components:

A. Physical Interaction-Based Task Decomposition & Skill Learning

Instead of treating demonstrations as monolithic tasks, DeCo preprocesses IL data based on gripper-object physical interactions.

Atomic Definition: A "full interaction" is defined as a cycle of gripper states: Open $\to$ Closed $\to$ Open. A "half interaction" is a single state change.
Decomposition: Original demonstrations are segmented into modular atomic tasks (e.g., "Open Drawer," "Place Item in Open Drawer") based on these interaction cycles.
Training: The IL model is trained on this decomposed dataset of atomic tasks, learning a policy $\pi$ to map language instructions and observations to specific 6-DoF end-effector poses and 1-DoF gripper states.

B. VLM-Guided Planning and Skill Scheduling

At inference time, DeCo acts as a planner for novel long-horizon tasks:

Input: The system takes a new natural language instruction and the current scene observation (RGB-D).
Reasoning: A Vision-Language Model (specifically GPT-4o) analyzes the instruction and scene to retrieve relevant atomic instructions from the pre-built library.
Scheduling: The VLM generates an ordered sequence of atomic skills. The IL model executes these skills sequentially.
Monitoring: DeCo monitors the robot's pose against the goal pose of the current skill. Once a match is detected (indicating task completion), it triggers the next skill in the sequence.

C. Spatially-Aware Skill Chaining

To ensure smooth transitions between atomic skills and prevent collisions:

Cost Map Generation: When a skill completes, DeCo generates a spatially-aware cost map (adapted from Voxposer) using the current goal pose, the predicted start pose of the next skill, and the scene point cloud.
Collision-Free Bridging: The system calculates intermediate "chaining poses" that bridge the gap between skills while avoiding obstacles.
Motion Planning: A short-horizon RRT-based planner executes the transition, ensuring the robot moves safely to the start pose of the next atomic skill without disrupting the learned policy distribution.

3. Key Contributions

DeCo Framework: A model-agnostic architecture that enables zero-shot generalization for compositional long-horizon tasks by decoupling task decomposition from the IL model.
Interaction-Based Decomposition: A novel method to convert raw demonstrations into reusable atomic skills based on physical gripper interactions, creating a consistent training signal.
DeCoBench: A new benchmark built on RLBench containing 10 atomic tasks and 12 novel long-horizon tasks (including cross-domain scenarios) specifically designed to evaluate compositional generalization.
Spatially-Aware Chaining: A module that solves the "handoff" problem between skills, ensuring collision-free and continuous motion in 3D space.

4. Experimental Results

The framework was evaluated on three representative multi-task IL models: RVT-2, 3DDA, and ARP.

Simulation Results (DeCoBench)

Baseline Failure: Without DeCo, baseline models achieved 0.00% success on novel long-horizon tasks, as they could not compose skills.
DeCo Success:
- RVT-2 + DeCo: Improved success rate from 0% to 66.67%.
- 3DDA + DeCo: Improved from 0% to 21.53%.
- ARP + DeCo: Improved from 0.14% to 57.92%.
Ablation: Using "full interaction" decomposition yielded better results than "half interaction," suggesting that complete interaction cycles provide clearer learning signals. Disabling the spatially-aware chaining module caused a significant drop in performance due to collisions.

Real-World Results

Setup: A Franka Emika Panda robot was trained on only 6 atomic tasks.
Performance: The DeCo-enhanced model successfully completed 9 novel long-horizon tasks in zero-shot.
Improvement: Achieved a 53.33% success rate on novel tasks, compared to 0% for the baseline RVT-2 model.
Robustness: The system handled minor perturbations (e.g., object replacement) but failed on significant displacements where the goal pose could not be reached, highlighting a need for future re-planning mechanisms.

5. Significance and Conclusion

This paper addresses a critical bottleneck in robotic learning: the inability of current models to generalize to complex, multi-step tasks without retraining.

Scalability: By decomposing tasks into atomic skills, DeCo allows robots to learn a small set of primitives and compose them into infinite variations of long-horizon tasks.
Practicality: The real-world validation demonstrates that this approach is not just a simulation artifact but viable for physical deployment.
Future Directions: The authors note limitations in handling significant object displacement (requiring closed-loop re-planning) and the current reliance on claw-like end-effectors. Future work aims to incorporate tactile feedback and dexterous control.

In summary, DeCo provides a robust, model-agnostic solution that bridges the gap between high-level semantic planning and low-level physical execution, significantly advancing the state of zero-shot generalization in 3D robotic manipulation.