SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

Imagine you are teaching a robot how to cook a complex meal. You don't just tell it "make dinner"; you show it a video of the whole process: chopping, frying, plating, and serving. In the world of robotics, this is called "embodied manipulation," where a machine learns to move its body in the real world based on what it sees. For a long time, scientists have tried to teach robots by feeding them massive amounts of these video demonstrations. The most popular methods, often called "Diffusion Policies" or "Vision-Language-Action" models, work like a super-smart student who memorizes the entire video. They are great at copying exactly what they've seen before. However, they hit a wall when faced with a new situation. If you ask a robot trained on "putting an apple in a bowl" to "put a lemon in a pot," it often freezes. It's like a student who memorized the answer key for one specific test but fails the moment the questions are rearranged. The problem is that these robots try to remember the whole movie at once, rather than understanding the individual scenes or "skills" that make up the story.

This is where a new idea called SkillMemo comes in. Instead of trying to memorize the entire video as one giant, unbreakable block, SkillMemo teaches the robot to break the video down into tiny, reusable "atomic skills"—like "grasp," "lift," or "place." It then stores these skills in a special, dynamic library (a memory bank) that the robot can search through while it's working. Think of it like a chef who doesn't just memorize a recipe for "stir-fry" but instead keeps a mental index of how to chop, how to sauté, and how to season. When the chef needs to make a new dish, they don't start from scratch; they pull the right "chopping" skill and the right "sautéing" skill from their memory and combine them instantly. This paper proposes that by giving robots this kind of "skill memory," they can become much better at handling new, unseen combinations of tasks, making them more flexible and human-like in their movements.

The Problem: Robots Who Can't "Mix and Match"

Current robot brains are impressive, but they have a specific weakness. They are trained on large datasets of human demonstrations, learning to predict the next move based on the current picture. However, these models often treat a task as a single, monolithic blob of data. If a robot learns to "pick up a cup and pour water," it learns that specific sequence. If you then ask it to "pick up a cup and pour juice," or "pick up a bowl and pour water," it might struggle because it hasn't seen that exact combination before. It lacks the ability to decompose the action into smaller, reusable parts. It's like a musician who can play a specific song perfectly but can't improvise a new melody using the same notes.

The authors argue that the solution isn't just to feed the robot more data (which is expensive and hard to get), but to change how the robot learns and remembers. They suggest that robots need to learn to identify and store "skills" separately, so they can mix and match them later.

The Solution: A Robot with a "Skill Library"

The paper introduces SkillMemo, a framework that acts like a smart librarian for robot skills. It works in two main stages:

1. Breaking the Movie into Scenes (Expert-Guided Trajectory Segmentation)
First, the system needs to figure out where one skill ends and another begins. Instead of asking a human to label every single move (which is tedious), SkillMemo uses a clever trick called a Mixture-of-Experts (MoE). Imagine a team of specialized workers (the "experts") inside the robot's brain. As the robot watches a demonstration, different workers take turns being "on duty." One worker might be great at "grasping," another at "moving," and another at "placing." The system learns to switch between these workers automatically. When the "grasping" worker is active, the robot knows it's in the "grasping" phase of the task. This happens without any human labels; the robot figures out the boundaries of the skills on its own by noticing patterns in the movement.

2. Storing and Retrieving Skills (Skill-Level Memory)
Once the robot has broken the task into these small, distinct skills, it stores them in a dynamic episodic memory bank. This isn't just a video file; it's a compact library of "how-to" instructions.

• The Key: The system saves a summary of what the situation looked like when the skill was used (e.g., "a bowl was nearby").
• The Value: It saves the specific instructions on how to perform that skill (the "gating coefficients" that tell the robot's brain which workers to activate).

When the robot faces a new task, it doesn't just guess. It looks at its current situation, searches its memory library for the most relevant skills, and pulls them out. It then combines these retrieved skills with its current plan. If it needs to "put a lemon in a pot," it might retrieve the "grasp lemon" skill from one memory and the "pour into pot" skill from another, fusing them together to create a new, successful action.

What the Experiments Showed

The researchers tested this idea in two ways: in computer simulations and on a real robot arm (a UR5e) in a lab.

In Simulation:
They tested SkillMemo on standard robot challenges, including the LIBERO benchmark (which tests how well robots can generalize to new tasks) and other environments like Push-T and Franka Kitchen.

• The Results: When they added SkillMemo to existing robot models (like Diffusion Policy and large Vision-Language-Action models), the robots got significantly better.
• The Numbers: On the LIBERO benchmark, the standard model π0.5 achieved a success rate of 96.8%. With SkillMemo, this jumped to 98.0%. While that might sound like a small number, in robotics, a 1.2% improvement is a massive deal.
• Generalization: The most exciting finding was how well the robots handled tasks they had never seen before. For example, if a robot was trained on "putting a strawberry in a bowl" and "putting a lemon on a plate," it could successfully figure out how to "put a strawberry on a plate" without ever being trained on that specific combination. The memory bank allowed it to reuse the "strawberry" skill and the "plate" skill in a new way.

In the Real World:
The team also tested the system on a real robot arm with five different tasks, such as putting a strawberry in a bowl or butter in a pot.

• The Results: SkillMemo consistently beat the standard Diffusion Policy. For the "Butter in Pot" task, the success rate went from 62.5% to 75.0%.
• Unseen Combinations: When they tested the robot on new combinations (like putting a lemon in a pot, which it hadn't seen during training), SkillMemo improved the success rate from 57.5% to 72.5%. This proved that the robot wasn't just memorizing; it was truly recombining skills.

What the Paper Rules Out and Confirms

The authors are careful to point out what their method is not. They argue against the idea that simply making the robot's brain bigger or training it on more raw data is the only solution. They show that without a structured way to remember and reuse skills, even powerful models struggle with new combinations.

They also confirm that their method works without needing humans to manually label every single "skill" in the training videos. The robot learns to identify these boundaries on its own. However, the paper is clear that these results are based on specific simulations and a limited set of real-world tasks. While the results are strong, they are not a magic bullet for every possible robot problem in existence. The success is measured in specific benchmarks, and the "real-world" tests were conducted in a controlled lab environment.

Why This Matters

The core idea of SkillMemo is that robots need to be more like us: capable of breaking complex actions down into smaller, reusable parts and remembering how to do them. By giving robots a "skill library" they can search and mix, we can help them handle the messy, unpredictable nature of the real world. Instead of being rigid copycats that fail when the script changes, these robots can become flexible problem-solvers, taking what they know and applying it to situations they've never encountered before. The paper suggests that this approach is a promising step toward making robots that can truly adapt to our world, rather than just repeating what they've been told.