Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation

Uni-Skill introduces a self-evolving framework that overcomes the limitations of fixed skill libraries by combining a hierarchical, automatically annotated skill repository (SkillFolder) with an adaptive planning module to enable few-shot, zero-shot generalizable robotic manipulation without manual intervention.

The Gist

Imagine you have a robot assistant. You want it to do a new chore, like "clean the messy desk."

The Old Way (The Problem):
Most robots today are like chefs who only know how to cook from a fixed, printed cookbook. If you ask them to make a dish that isn't in the book (like "wipe the table with a sponge"), they freeze. They can't improvise. Even if you try to teach them a new recipe on the spot, they need a human to stand there, hold their hand, and show them exactly how to move their arm for every single step. This is slow, expensive, and limits the robot to only what humans have explicitly taught it beforehand.

The New Way (Uni-Skill):
The paper introduces Uni-Skill, which is like giving the robot a super-intelligent librarian and a giant, self-updating library of how-to videos.

Here is how it works, broken down into three simple parts:

1. The "Skill-Aware" Brain (The Detective)

When you give the robot a command, Uni-Skill doesn't just blindly try to execute it. First, it acts like a detective checking its own toolbox.

• The Question: "Do I have the right tools to clean this desk?"
• The Realization: It might say, "I know how to pick up things and place them, but I don't have a specific 'wipe' skill yet."
• The Fix: Instead of giving up, it invents a description for the missing skill: "Okay, I need a new skill called 'Wipe Table' that involves holding a cloth and moving it in circles." It essentially writes its own job description for the new task.

2. The "SkillFolder" (The Giant Library)

This is the magic part. The robot doesn't need a human to teach it the new skill. Instead, it goes to SkillFolder.

• What is it? Imagine a massive library containing over 10,000 hours of unorganized, messy videos of robots (and humans) doing all kinds of things.
• The Organization: Before, this library was a chaotic mess. Uni-Skill organizes it like a hierarchical filing system (inspired by how dictionaries group words). It sorts videos not just by "cleaning," but by specific nuances like "wiping with a sponge," "wiping with a cloth," or "wiping a table."
• The Search: When the robot needs to learn "Wipe Table," it searches this library. It finds a video of someone wiping a table. It doesn't just watch the video; it extracts the essence of the movement.

3. The "Few-Shot" Learner (The Mimic)

Once the robot finds a relevant video in its library, it doesn't need to watch the whole thing or have a human guide it.

• The Analogy: Think of it like learning to ride a bike. You don't need a coach to hold your hand for every pedal stroke. You just need to see one good example of someone riding, understand the balance and the path, and then try it yourself.
• The Execution: The robot looks at the "Wipe Table" video it found. It learns the pattern (move in circles) and the constraints (keep the cloth touching the surface). It then uses this "mental blueprint" to figure out exactly how to move its own arm in your specific kitchen, even if your table is a different shape or color.

Why is this a big deal?

• No More "Freezing": If you ask the robot to do something it's never seen before, it doesn't crash. It checks its library, finds a similar example, and figures it out.
• Zero-Shot Learning: It can do new tasks without you ever showing it a demonstration in that specific moment. It learns from its massive offline library.
• Real-World Results: The researchers tested this in simulations and with a real robot arm. When asked to do tricky, new tasks (like folding a cloth or closing a specific type of drawer), Uni-Skill was 31% to 34% more successful than the best existing methods.

In a nutshell:
Uni-Skill turns a robot from a rigid machine that only follows a fixed script into a curious, adaptable apprentice. It can look at a new problem, realize what it's missing, look up a similar example in its massive video library, and figure out how to do the job on its own. It's the difference between a robot that can only play "Chopsticks" on the piano and one that can listen to a song once and then improvise its own version.

Technical Summary

1. Problem Statement

Current robotic manipulation systems face significant limitations in generalizing to novel, long-horizon tasks:

• Fixed Skill Libraries: Existing skill-centric approaches rely on predefined sets of skills (APIs). If a required skill (e.g., "fold cloth") is not in the library, the system fails.
• Manual Intervention: Expanding these libraries typically requires manual annotation of demonstrations or waypoint labeling for every new task, which is inefficient and does not scale.
• Unstructured Data Gap: While vast amounts of unstructured robotic videos exist, they lack the necessary semantic links and annotations to be directly used as skill demonstrations.
• Lack of Adaptability: Current frameworks cannot identify "skill gaps" or autonomously evolve their capabilities to bridge the gap between high-level instructions and low-level execution without human supervision.

2. Methodology: Uni-Skill Framework

The authors propose Uni-Skill, a unified framework that integrates Skill-Aware Planning and Automatic Skill Evolution. It shifts the paradigm from static skill sets to a self-augmenting, evolving repository.

A. Skill-Aware Planning Module

This module enables the system to handle instructions that exceed its current capabilities.

• Sufficiency Discriminator ($E$): Evaluates whether the existing basic skill library ($L_{base}$) is sufficient to execute a given multimodal instruction ($I_t, O_t$).
• Skill Generator ($G$): If the discriminator identifies a gap, it autonomously synthesizes descriptions for new, supplementary skills ($L_{ext}$).
• Planner ($P$): Generates executable policy code based on the combined library ($L_{base} \cup L_{ext}$).
• Implementation: Utilizes a Vision-Language Model (VLM) trained on 106K runtime code samples to perform these tasks, allowing for code-based output that aligns with robotic APIs.

B. Automatic Skill Evolution Module

This module grounds the newly generated skill descriptions into executable actions using large-scale unstructured video data, eliminating the need for deployment-time demonstrations.

1. Automatic Skill Annotation Pipeline:

   • Uses a VLM (Gemini-2.0-Flash) to process raw robotic videos (e.g., from the DROID dataset).
   • Extractor: Generates procedural action plans from keyframes.
   • Descriptor: Converts plans into task-oriented skill descriptions, checking against existing skills to avoid redundancy.
   • Aligner: Uses a "Number-It" strategy to temporally align skill descriptions with specific video segments.
   • Result: Converts 350 hours of unstructured video into 10,000+ annotated skill traces.

2. SkillFolder (Hierarchical Repository):

   • Inspired by VerbNet and the ImageNet/WordNet relationship, the authors construct a 4-layer hierarchical taxonomy:
     • Layer 1 ($N_1$): Abstract VerbNet classes (e.g., wipe-manner).
     • Layer 2 ($N_2$): Distinct verb instances (e.g., scrub, wipe, rinse).
     • Layer 3 ($N_3$): Skill descriptions aligned with object-centric interactions (e.g., wipe_table).
     • Layer 4 ($N_4$): Fine-grained, visually grounded video slices (examples).
   • This structure allows for efficient retrieval of semantically relevant examples for new skills.

3. Few-Shot Skill Implementation:

   • When a new skill is requested, the system retrieves relevant examples from SkillFolder.
   • Semantic Constraints: Extracts high-level constraints (contact points, waypoints) from examples using VLMs.
   • Spatial Trajectory: Projects 6-DoF poses from examples onto the target scene to generate 2D waypoints, which are then lifted to 3D using depth information.
   • Orientation Transfer: Uses local frames to transfer rotation patterns from source examples to the target scene, ensuring physically feasible execution.

3. Key Contributions

• Self-Evolving Framework: Uni-Skill is the first framework to autonomously detect skill gaps, generate new skill descriptions, and implement them using a structured, self-augmented repository without manual intervention.
• SkillFolder Dataset: A novel, VerbNet-inspired hierarchical repository containing 106 VerbNet classes, 1,659 unique skill formulations, and over 10,000 automatically annotated skill traces derived from unstructured videos.
• Zero-Shot Generalization: The system achieves state-of-the-art performance on novel tasks by leveraging retrieved examples for few-shot inference, removing the dependency on deployment-time demonstrations.
• Unified Code-Based Planning: Integrates planning and skill generation into a single VLM-based pipeline that outputs executable code, bridging the gap between high-level reasoning and low-level control.

4. Experimental Results

Experiments were conducted in both simulation (RLBench) and real-world settings (Franka Emika robot).

• Simulation (RLBench):

  • Out-of-Distribution Tasks: On 10 tasks requiring skills outside the basic set, Uni-Skill achieved a 41% success rate, significantly outperforming MOKA (10%) and CaP (1%).
  • Improvement: Uni-Skill outperformed the state-of-the-art visual prompting method MOKA by 31.0% on zero-shot tasks.
  • Long-Horizon Tasks: Showed a 20.0% improvement on complex long-horizon tasks.

• Real-World Deployment:

  • On 8 diverse real-world tasks (including "Fold Cloth" and "Stir Blocks"), Uni-Skill achieved an average success rate of 73%, compared to 39% for MOKA and 0% for CaP (which failed on unseen skills).
  • Specifically, on "Fold Cloth," Uni-Skill achieved 70% success vs. 20% for MOKA.

• Ablation Studies:

  • Disabling the skill updating mechanism caused the system to fail on most novel tasks.
  • Both semantic constraints and spatial trajectory references were found critical; removing semantic constraints hurt contact-sensitive tasks (e.g., Revolute), while removing trajectory references hurt precise spatial tasks (e.g., Unplug).

5. Significance

• Paradigm Shift: Moves robotic learning from "manual curation of skills" to "automatic acquisition and evolution" from unstructured data.
• Scalability: Demonstrates that large-scale, noisy robotic videos can be transformed into structured, high-quality training resources using VLMs, drastically reducing the cost of skill acquisition.
• Generalization: Proves that robots can handle completely novel instructions and environments by dynamically composing and retrieving skills, a crucial step toward truly general-purpose robotic assistants.
• Robustness: The system includes self-correcting mechanisms for failure modes (planning, grounding, execution), further enhancing reliability in real-world scenarios.