💻 computer science

Learning New Tasks via Reusable Skills: Skill-Compositional Experts for Embodied Continual Learning

This paper proposes SCE, a Skill-Compositional Experts framework that mitigates catastrophic forgetting in Embodied Continual Learning by decomposing tasks into reusable skills via Compositional Skill Grounding and coordinating their execution and transitions through Dual Execution-and-Transition Experts.

Original authors: Shuaike Zhang, Shaokun Wang, Haoyu Tang, Jianlong Wu, Liqiang Nie

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuaike Zhang, Shaokun Wang, Haoyu Tang, Jianlong Wu, Liqiang Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do chores around the house. First, you teach it how to open a drawer. Then, you teach it how to pick up a bowl. Finally, you teach it how to put that bowl inside the drawer.

In the world of robotics, this is called Embodied Continual Learning. The robot has to keep learning new tasks without forgetting how to do the old ones.

The Problem: The "Drifting" Robot

The paper explains that most robots have a serious memory problem. When you teach them a new trick, their internal "brain" (specifically, the way they process visual and language data) starts to shift or "drift" to accommodate the new task.

Think of it like a musician who learns a new song. If they practice the new song too much, they might start playing the old songs slightly wrong, mixing in the new rhythm. In a robot, this is dangerous because it's not just playing a song; it's moving physical arms. If the robot's internal map drifts, a small mistake in one step can get bigger and bigger as it tries to finish a task, eventually causing it to fail completely. This is called catastrophic forgetting.

The Solution: The "Lego Box" Approach (SCE)

The authors propose a new framework called SCE (Skill-Compositional Experts). Instead of teaching the robot to see every new task as a completely brand-new, giant instruction, they teach it to break everything down into small, reusable Lego bricks (skills).

Here is how SCE works, using two main tools:

1. The "Skill Organizer" (CSG)

Imagine you have a box of instructions. The Compositional Skill Grounding (CSG) module is like a smart librarian. It looks at a long video of a robot doing a complex task (like "put the bowl in the drawer") and chops it up into tiny, reusable chunks:

Chunk A: "Grab the bowl."
Chunk B: "Lift the bowl."
Chunk C: "Open the drawer."
Chunk D: "Place the bowl."

It saves these chunks in a Skill Base (a digital library). Now, when the robot needs to learn a new task, it doesn't start from scratch. It just grabs the existing "Grab" and "Place" bricks from the library and snaps them together in a new order.

2. The "Dual-Expert Brain" (DETE)

Once the robot has its library of Lego bricks, it needs a brain to decide how to use them. This is the Dual Execution-and-Transition Experts (DETE) module. It has two specialized helpers working together:

The "Doer" (Execution Expert): This helper is great at performing a specific skill perfectly. If the robot needs to "Grab the bowl," this expert takes over and ensures the robot grabs it exactly right, just like it did the first time. This prevents the robot from "forgetting" how to grab.
The "Switcher" (Transition Expert): This helper is the master of transitions. It knows how to smoothly move from "Grabbing" to "Lifting." It handles the messy middle part where one skill ends and another begins.

These two experts work together. The "Doer" keeps the individual skills stable, while the "Switcher" makes sure the robot doesn't stumble when changing from one action to the next.

Why This Matters

The paper tested this on a computer simulation (using the LIBERO benchmark) and on a real physical robot arm.

In the simulation: The new method (SCE) was much better at remembering old tasks while learning new ones compared to other methods. It forgot much less.
In the real world: When they tried it on a real robot arm, it worked even better. The robot could successfully complete a sequence of tasks it had learned over time, whereas other methods made the robot fail the old tasks after learning new ones.

The Big Picture

The paper claims that by treating robot tasks like combinations of reusable skills (like building with Lego) rather than one giant, unchangeable block of memory, robots can learn continuously without losing their mind. The "Doer" keeps the skills sharp, and the "Switcher" keeps the flow smooth, preventing the robot's internal map from drifting off course.

In short: SCE teaches robots to be like master chefs who have a set of perfect basic techniques (chopping, frying, boiling). Instead of memorizing a whole new recipe from scratch every time, they just mix and match these perfect techniques to create new dishes, ensuring they never forget how to chop an onion.

Technical Summary: Skill-Compositional Experts for Embodied Continual Learning

Problem Definition

The paper addresses Embodied Continual Learning (ECL), a setting where robots must sequentially acquire new manipulation tasks while retaining previously learned behaviors under closed-loop control. Unlike conventional continual learning (CL) where data is static, ECL involves dynamic environments where observations evolve based on the robot's actions.

The core challenge identified is catastrophic forgetting, which is exacerbated in ECL by feature drift. As Vision-Language-Action (VLA) models learn new tasks, their feature representations drift away from those associated with old tasks. Under closed-loop control, this drift is not merely a static performance drop; it propagates through sequential decision-making, causing errors to compound and leading to the degradation of previously learned behaviors. Existing methods often treat tasks as isolated units or focus on skill discovery without explicitly organizing skills for coherent composition, leading to inconsistent reuse and amplified drift.

Methodology: The SCE Framework

The authors propose SCE (Skill-Compositional Experts), a framework designed to enable continual task learning through the composition of reusable skills. The architecture consists of two primary components integrated into a pretrained VLA model:

1. Compositional Skill Grounding (CSG)

CSG is responsible for decomposing task demonstrations into reusable skills and organizing them into a Skill Base. It operates in two phases:

State-Aware Task Decomposition: Using robot-state dynamics (specifically gripper state changes and end-effector motion trend changes), CSG partitions raw task demonstrations into temporally coherent segments.
VLM-Grounded Skill Induction: A Vision-Language Model (VLM) grounds these segments against the current skill base. If a segment matches an existing skill, it is assigned that skill; otherwise, a new reusable skill is induced and added to the base. This process generates skill annotations ( $s$ ) for the training data, allowing the model to learn a structured skill space.

2. Dual Execution-and-Transition Experts (DETE)

DETE augments the VLA action decoder with two complementary expert branches to support skill composition:

Execution Expert Branch: Focuses on modeling skill-specific action patterns. It uses a router to select the most relevant expert from a pool based on the current skill distribution, ensuring stable execution of individual skills. This confinement of updates to specific experts reduces interference with unrelated skills.
Transition Expert Branch: Focuses on modeling cross-skill transition patterns. Since skill changes often occur over temporal intervals rather than instantaneously, this branch captures dependencies between preceding and subsequent skills. It uses a transition-aware router that incorporates the skill distribution to model the ambiguity during skill switching.
Adaptive Fusion: An Adaptive Fusion Module dynamically combines the outputs of the Execution and Transition branches. It computes a balancing coefficient ( $\alpha$ ) based on the transition-aware routing feature. When the skill distribution is concentrated on a single skill, the model emphasizes execution; near skill boundaries, it shifts weight toward the transition branch to ensure coherent behavior.

The model is optimized using a combined loss function: the original flow-matching loss for action generation and an auxiliary cross-entropy loss to supervise the skill distribution.

Key Contributions

SCE Framework: The proposal of a Skill-Compositional Experts framework that enables continual learning by composing reusable skills, addressing the specific challenges of feature drift in closed-loop ECL.
CSG and DETE Modules:
- CSG: A mechanism to decompose demonstrations into reusable skills and maintain a structured skill base.
- DETE: A dual-expert architecture that separates skill-specific execution from cross-skill transitions, allowing for stable execution and coherent switching.
Empirical Validation: Extensive experiments demonstrating that SCE effectively mitigates feature drift and improves retention compared to state-of-the-art baselines.

Experimental Results

The authors evaluated SCE on both simulated and real-world benchmarks:

Simulation (LIBERO Benchmarks):
- On LIBERO-Goal, SCE achieved a Final Success Rate (SR) of 83.4% and a New Task Forgetting (NBT) of 4.3%, significantly outperforming baselines like Task-MoE (64.8% SR, 15.8% NBT).
- On LIBERO-Long, SCE achieved a Final SR of 73.4% with an NBT of 0.5%, demonstrating superior stability over long horizons.
- Metrics used included Forward Transfer (FWT), NBT, Area Under Curve (AUC), and Final SR.
Real-World Manipulation:
- Experiments on a 7-DoF Mercury B1 robot with three sequential tasks (Box-to-Board, Board-to-Bowl, Box-to-Bowl) showed SCE achieving a 83.3% Final SR and 0.0% NBT, outperforming the Task-MoE baseline (56.7% SR, 10.0% NBT).
Feature Drift Analysis:
- Analysis revealed that while standard sequential training caused significant drift in action features (up to 6.9× larger than VLM features), SCE substantially reduced this drift.
- Shared-anchor rollouts demonstrated that SCE maintained a much smaller simulator-state gap from reference behaviors under closed-loop control compared to baselines.
Ablation Studies:
- Removing CSG resulted in a drop in Final SR from 83.4% to 70.0%, highlighting the necessity of explicit skill organization.
- Removing either the Execution or Transition expert branch degraded performance, confirming that both skill-specific execution and cross-skill transition modeling are complementary and necessary.

Significance and Claims

The paper claims that SCE provides a robust solution to the stability-plasticity trade-off in embodied continual learning. By structuring knowledge as reusable skills and explicitly modeling the transitions between them, the method prevents the accumulation of feature drift that typically degrades closed-loop performance. The authors assert that their approach enables robots to retain previously learned behaviors more effectively than task-level or unstructured skill-level methods, making it a viable step toward long-term adaptability in real-world robotic deployment. The work suggests that future directions could involve extending this compositional approach to larger-scale and more open-ended skill repertoires.