💻 computer science

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

This paper presents AtomSkill, a novel framework that learns a semantically aligned atomic skill space through contrastive alignment and keypose imagination to enable robust, generalizable multi-task robotic manipulation by decomposing behaviors into reusable, progress-aware skills.

Original authors: Yihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi, Jingya Wang

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi, Jingya Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to do chores. If you just show it a video of someone cleaning a kitchen, the robot might try to memorize every single muscle movement. But if you ask it to clean a different kitchen with the dishes in a different spot, it gets confused and fails. It's like trying to memorize a specific route to a friend's house; if the friend moves, you're lost.

The paper introduces AtomSkill, a new way to teach robots that is more like teaching a human the concepts of chores rather than just the specific movements.

Here is how it works, broken down into simple ideas:

1. The Problem: The "Over-Memorizing" Robot

Current robots often struggle when faced with many different tasks. They either get confused by the noise in the data or they try to do everything at once, causing a "traffic jam" in their brain where one task interferes with another. They lack a library of reusable "building blocks."

2. The Solution: The "Atomic Skill" Library

The authors propose breaking complex tasks down into tiny, reusable chunks called Atomic Skills.

The Analogy: Think of a Lego set. Instead of trying to build a whole castle from a single, giant, unbreakable block, you have small bricks: "a wall," "a window," "a door."
How AtomSkill does it: It takes a long video of a robot doing a task and chops it up into these small, meaningful pieces (like "grasp the spoon" or "place the lid").
The "Smart" Part: It uses a giant AI brain (a Vision-Language Model) to read the video and label these pieces with human words. It doesn't just see "arm moves up"; it understands "grasping." This ensures that if the robot learns to "grasp" a cup in one task, it knows that "grasping" a spoon is the same type of skill, even if the motion looks slightly different.

3. The Secret Sauce: "Keypose Imagination"

This is the most creative part of the paper. When the robot is learning a skill, it doesn't just learn how to move; it also learns to imagine the finish line.

The Analogy: Imagine you are walking through a dark forest. A normal robot might just take one step at a time, hoping it doesn't fall. An AtomSkill robot is like a hiker who knows exactly where the next campsite (the "keypose") is.
How it works: As the robot performs a skill (like "grasp"), it simultaneously predicts what the robot's hand will look like when that skill is finished.
Why it helps: This acts as a progress monitor. The robot keeps moving until its hand matches the "imagined finish line." Once it hits that mark, it knows, "Okay, this job is done," and it smoothly switches to the next skill without needing a human to tell it when to stop.

4. Putting It All Together: The "Diffusion Sampler"

When the robot needs to do a new, complex task (like "tidy up the pot"), it doesn't panic.

The Analogy: Think of a chef planning a meal. They don't invent a new recipe from scratch every time. They pull "grasp," "lift," and "place" from their mental cookbook and string them together.
How it works: AtomSkill uses a "diffusion sampler" (a fancy math tool that generates possibilities) to pick the right sequence of these atomic skills from its library. It then executes them one by one, using the "Keypose Imagination" to know exactly when to switch from one skill to the next.

The Results

The researchers tested this in computer simulations and with real robots doing things like:

Wiping a whiteboard.
Sweeping trash into a dustpan.
Unplugging a charger.
Putting a flower in a vase (using two robot arms working together).

The Outcome: AtomSkill consistently beat other top methods. It was better at handling tasks where the robot had to move through multiple stages and figure out exactly where to stop. It learned faster and made fewer mistakes because it understood the meaning of the actions, not just the raw movements.

In a nutshell: AtomSkill teaches robots to think in "chunks" of meaning rather than a blur of motion, and it gives them a mental map of where each chunk ends, allowing them to chain complex tasks together smoothly.

Technical Summary: AtomSkill

Problem Statement

Scaling imitation learning (IL) to diverse multi-task robotic manipulation remains a significant challenge. While recent methods like Diffusion Policy (DP) and ACT have demonstrated success in single-task settings, they struggle with fundamental issues in multi-task scenarios: suboptimal expert demonstrations, trajectory noise, and behavioral multi-modality. These issues often lead to policies that fail to generalize or suffer from destructive interference across tasks.

Existing skill-based approaches attempt to decompose behaviors into reusable abstractions but face two primary limitations:

Linguistic Bias: Methods like LISA and SkillDiffuser learn linguistic abstractions from language and visual inputs but often lack smooth alignment with the action space, making it difficult to disentangle task-specific behaviors or model multimodal actions.
Lack of Semantic Alignment: Action-only methods (e.g., Discrete Policy, QueST, STAR) learn skills directly from raw action sequences using vector quantization (VQ). However, these abstractions are often driven by low-level motion regularities rather than semantic equivalence. Consequently, motions with similar intents but different task contexts are encoded as separate latent tokens, preventing effective cross-task alignment and limiting generalization.

Methodology: AtomSkill

The authors propose AtomSkill, a framework designed to learn a semantically aligned Atomic Skill Space from demonstrations and enable robust long-horizon execution through keypose imagining. The framework consists of three core components:

1. Semantic Skill Discovery with VLM

The framework first segments expert demonstration trajectories into non-overlapping, semantically coherent sub-trajectories (atomic skills).

Segmentation: Trajectories are split based on gripper-state keyframes. A change in gripper state (e.g., open to closed) serves as a robust indicator of the end of one skill and the start of another.
Labeling: A large Vision-Language Model (VLM) is employed to analyze each segment and assign a semantic skill label (e.g., "grasp," "place") and a natural language description. This grounds the skill boundaries in semantic structure rather than just temporal or low-level motion patterns.

2. Atomic Skill Learning with Semantic Contrastive Alignment

The core learning module structures the latent skill space using a VQ-VAE architecture enhanced with contrastive learning.

Skill Quantization: A skill encoder compresses variable-length sub-trajectories into fixed-length continuous embeddings, which are then mapped to discrete tokens via a learnable codebook (Vector Quantization).
Semantic Contrastive Loss: To ensure the discrete tokens represent semantically meaningful skills, the authors introduce a contrastive objective with two terms:
- Temporal Consistency ( $L_{temp}$ ): Encourages embeddings sharing the same quantized token and relative position within a sequence to be close, ensuring structural coherence.
- Semantic Awareness ( $L_{skill}$ ): Encourages embeddings sharing the same semantic label (from the VLM) to cluster together, even if they originate from different tasks. This enforces that functionally similar skills (e.g., "grasp" in different contexts) share the same latent representation.
Action Decoder with Keypose Imagining: The decoder ( $\psi_\theta$ $ψ_{θ}$ ) takes observations and the current skill token as input. Crucially, it jointly predicts:
1. The immediate action chunk (short-term control).
2. The terminal keypose ( $a_{keypose}$ ), representing the goal state of the current skill.

3. Inference with Atomic Skill Priors

During execution, the system employs a diffusion-based sampler and a keypose-driven transition strategy.

Atomic Skill Prior: A diffusion model ( $\rho_\theta$ ) samples high-level skill embeddings conditioned on the task instruction and current observation. This sampler is invoked only when a new skill is required.
Action Chunking with Keypose: The policy generates a chunk of actions. The transition to the next skill is triggered autonomously when the predicted subsequent action becomes sufficiently close to the predicted terminal keypose. This eliminates the need for manual heuristics or additional classifiers to determine skill termination, enabling smooth and robust skill chaining.

Key Contributions

AtomSkill Framework: A novel multi-task imitation learning framework that learns and leverages a structured, semantically aligned Atomic Skill Space.
Semantically Grounded Skill Library: A paradigm that partitions demonstrations using gripper-state detection and VLM annotation, employing a contrastive objective to create a compact, reusable skill codebook that aligns semantic consistency with temporal coherence.
Keypose-Conditioned Action Generation: An action decoder that jointly predicts terminal keyposes and immediate actions. This mechanism facilitates simultaneous reasoning about long-horizon motion goals and fine-grained control, enabling robust skill chaining without manual intervention.

Experimental Results

The authors evaluated AtomSkill on both simulation (RLBench) and real-world environments (single-arm and dual-arm tasks).

Simulation (RLBench): Across 12 diverse tasks, AtomSkill achieved an average Success Rate (SR) of 74.2%, outperforming state-of-the-art baselines including MT-ACT (68.6%), DP (36.9%), and QueST (32.8%). It showed particular strength in tasks requiring spatial localization and multi-stage coordination.
Real-World Single-Arm: AtomSkill achieved an Average Task Progress (ATP) of 0.93 and SR of 90.0%, significantly outperforming baselines (e.g., ACT at 0.73 ATP).
Real-World Dual-Arm: In complex bimanual tasks (e.g., "Pot Tidy Up"), AtomSkill achieved the highest performance (0.60 ATP, 46.7% SR) despite being trained from scratch with limited data (100 demonstrations per task), whereas baselines like RDT (which relies on large-scale pretraining) struggled significantly.
Ablation Studies: Removing either the temporal or semantic contrastive loss degraded performance, confirming their complementary roles. Similarly, removing keypose prediction reduced success rates, particularly in localization-heavy tasks, validating its role in spatial grounding.

Significance and Claims

The paper claims that AtomSkill addresses the critical gap in multi-task imitation learning by moving beyond low-level motion regularities to learn semantically aligned atomic skills. By explicitly grounding skill boundaries in semantic structure via VLMs and enforcing semantic consistency through contrastive learning, the method achieves a skill space that is both compact and highly transferable across tasks.

The integration of keypose imagining is highlighted as a key innovation for long-horizon execution. It allows the policy to reason about the terminal state of a skill, enabling autonomous and smooth transitions between skills without relying on brittle, hand-crafted termination heuristics.

The authors acknowledge a limitation: defining atomic skill granularity for continuous tasks without clear physical transitions (e.g., spreading sauce) may introduce ambiguity. However, the results demonstrate that for tasks with distinct interaction phases, AtomSkill provides a robust solution for scalable, multi-task robot manipulation.