💻 computer science

One-Shot Cross-Geometry Skill Transfer through Part Decomposition

This paper proposes a one-shot cross-geometry skill transfer method that decomposes objects into semantic parts and leverages data-efficient generative shape models to accurately align interaction points, enabling robots to generalize skills to objects with unfamiliar shapes in both simulated and real environments.

Original authors: Skye Thompson, Ondrej Biza, George Konidaris

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Skye Thompson, Ondrej Biza, George Konidaris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to pour tea from a teapot into a mug. You show it once: "Grab the handle, tilt the spout, and pour."

If the robot is smart, it should be able to do this with any teapot it sees later, even if that teapot is shaped differently, has a longer handle, or a shorter spout. But here's the problem: most robots today are like students who memorized the answer key for one specific test. If you give them a slightly different test (a different-shaped teapot), they get confused and fail. They try to apply the exact same hand movements to a completely different shape, leading to spills or crashes.

This paper introduces a new way to teach robots that is more like teaching them to understand the "parts" of an object rather than just memorizing the whole thing.

Here is the breakdown of their idea using simple analogies:

1. The Problem: The "Monolith" Mistake

Imagine trying to fit a square peg into a round hole, but you are only allowed to look at the peg as one giant, solid block of clay. If the clay changes shape slightly, you have no idea how to adjust because you were taught to treat it as one single, unchangeable unit.

Current robots often do this. They look at a teapot as one big blob. When they see a new teapot with a weirdly long spout, they get lost because their "blob" model doesn't match the new reality.

2. The Solution: The "Lego" Approach (Part Decomposition)

The authors suggest we stop looking at objects as giant blobs and start looking at them like Lego sets.

Old Way: "This is a Teapot."
New Way: "This is a Teapot made of a Body, a Handle, a Spout, and a Lid."

By breaking the object down into these independent parts, the robot can understand that the Handle is for holding, the Spout is for pouring, and the Body is for holding the water. These parts have specific jobs, regardless of how the whole teapot looks.

3. How It Works: The "Shape-Shifting" Translator

Once the robot breaks the object into parts, it uses a clever trick called Generative Shape Warping.

Think of it like a digital tailor or a 3D printer.

The robot looks at the new teapot and identifies its parts (e.g., "Ah, there's the handle").
It has a "mental library" of what handles usually look like.
It uses this library to "stretch" and "warp" the new handle to match the shape of the handle from the demonstration.
It does this for every part individually.

This is much better than trying to stretch the whole teapot at once. If the new teapot has a handle that is twice as long, the robot just stretches the "handle part" of its mental model. It doesn't get confused by the rest of the teapot.

4. The Secret Sauce: "Relational Descriptors" (The Glue)

There is a catch. If you just stretch the handle and the spout separately, they might end up in the wrong places relative to each other. You might end up with a handle on the bottom and a spout on the top!

To fix this, the robot uses Relational Descriptors.

Analogy: Imagine you are assembling a puzzle. You know the "Sky" piece goes next to the "Tree" piece.
The robot learns that the "Spout" is always connected to the "Body" in a specific way. Even if the spout is long or short, the robot knows, "Okay, the spout must point away from the body."

This ensures that when the robot warps the parts to fit the new object, the parts stay in the correct relationship to one another.

5. The Result: One-Shot Learning

The most impressive part is that this robot only needs to see one single demonstration to learn the skill.

Scenario: You show the robot how to pour from a small, round teapot.
Test: You give it a tall, rectangular watering can (which looks nothing like the teapot).
Outcome: The robot breaks the watering can into parts (Body, Spout, Handle). It realizes the "Spout" part is the key for pouring. It warps its knowledge of the small teapot's spout to fit the watering can's spout. It successfully pours the water.

Why This Matters

This method is like teaching a robot the grammar of objects instead of just memorizing vocabulary.

Old Robots: Memorized "Teapot = Pour." (Fails on new shapes).
New Robot: Understands "Handle = Hold, Spout = Pour, Body = Hold Liquid." (Works on teapots, watering cans, jugs, and weird alien containers).

The researchers tested this in computer simulations and on a real robot arm. They found that this "Lego" approach allowed the robot to succeed with a much wider variety of strange and new objects than previous methods, all while only needing to see the task performed once.

In short: Instead of trying to memorize the shape of every single object in the world, this method teaches the robot to recognize the parts of an object and how those parts fit together, allowing it to adapt to anything it encounters.

1. Problem Statement

The core challenge addressed is robotic skill generalization: enabling a robot to execute a manipulation skill demonstrated on a specific object to a novel object with a significantly different geometry, using only a single demonstration (one-shot learning).

Existing approaches (e.g., whole-object shape warping or neural descriptor fields) often fail when encountering objects with unfamiliar shapes because:

They treat objects as monolithic entities, failing to disentangle the geometry of specific functional parts (e.g., a handle vs. a spout) from the overall object shape.
They rely on global priors that are difficult to define for the full range of shape variations, leading to misalignment of interaction points.
They struggle when the spatial relationships between parts change (e.g., a teapot with a long spout and short handle vs. one with a short spout and long handle).

2. Methodology

The authors propose Part-Based Shape Warping (PSW), a method that decomposes objects into semantic parts to improve feature transfer and skill adaptation. The pipeline consists of four main stages:

A. Part Decomposition

Instead of modeling an object as a single mesh, the system decomposes objects into a fixed set of constituent semantic parts (e.g., a mug $\rightarrow$ cup + handle; a teapot $\rightarrow$ body + spout + handle + lid).

Input: A demonstration involving objects $A$ and $B$ .
Process: The objects are segmented into parts $M$ and $N$ . Interaction points are identified between specific pairs of parts (e.g., mug handle to rack peg) rather than globally.

B. Generative Shape Modeling & Warping

For each part category, a generative shape model is trained using a small set of 5–10 pose-aligned point clouds.

Latent Space: The model uses Principal Component Analysis (PCA) on the deformation space of the parts to create a low-dimensional latent vector.
Inference: Given a partial point cloud of a novel object part, the model infers the latent vector to reconstruct the full geometry and predict the positions of interaction points on the new object.
Advantage: This allows accurate transfer of interaction points even with occlusion or partial views, as the model learns the "shape manifold" of that specific part.

C. Relational Descriptors (Symmetry Breaking)

A key innovation is addressing the ambiguity caused by decomposing objects (e.g., a cylindrical cup has rotational symmetry, making it hard to orient the handle correctly).

Mechanism: The authors introduce relational descriptors (binary labels) during training. These labels encode the spatial relationship between parts (e.g., "points on the spout that are close to the body").
Optimization: During inference, the Chamfer distance metric is modified to only match points sharing these relational labels. This breaks symmetry and ensures the reconstructed geometry aligns correctly with the observed object (e.g., ensuring the spout points away from the body).

D. Heuristic Optimization for Skill Transfer

The system autonomously constructs an optimization objective to find the correct transformation ( $T_{A'B'}$ ) for the novel objects.

Relevance Selection: The system evaluates which part relationships are critical for the specific task (e.g., for placing a mug on a rack, the handle-to-peg relationship is critical; the lid is not).
Optimization: It optimizes the global transformation to minimize the cost function (alignment error) only for the relevant part pairs, ignoring irrelevant geometry. This prevents errors caused by conflicting geometric features.

3. Key Contributions

Part-Based Decomposition for Transfer: Demonstrates that decomposing objects into semantic parts allows for more accurate feature transfer and shape reconstruction than monolithic modeling, especially when part relationships vary.
Relational Descriptors: Introduces a method to break geometric symmetries and improve reconstruction accuracy by encoding inter-part relationships directly into the optimization objective.
One-Shot Learning: Achieves successful skill transfer with only one demonstration and 5–10 object examples per part category, significantly reducing data requirements compared to neural descriptor fields (which often require hundreds of examples).
Autonomous Objective Construction: The method automatically identifies which part relationships are salient for a task and optimizes the policy based on those specific relationships, rather than relying on a fixed global heuristic.

4. Experimental Results

The method was evaluated in both simulation (PyBullet) and on a real-world robot (Franka Emika Panda) across three tasks:

Tasks: Placing a mug on a rack, placing a bowl on a mug, and pouring tea from a teapot.
Baselines: Compared against Whole Object Interaction Warping (IW) and Relational Neural Descriptor Fields (R-NDF).

Key Findings:

Simulation:
- Mug on Rack: PSW achieved 78% success rate vs. 64% (IW) and 42% (R-NDF).
- Bowl on Mug: PSW achieved 84% success rate vs. 79% (IW) and 43% (R-NDF).
- PSW succeeded on a wider variety of geometries, particularly those with narrow gaps or complex part arrangements where global warping failed due to mesh collisions.
Real Robot:
- PSW significantly outperformed IW in success rates (e.g., 23/27 vs. 11/27 for Mug/Rack).
- Cross-Category Generalization: The method successfully transferred skills from a teapot to a watering can (which shares the same functional parts: spout, handle, body), demonstrating robustness to objects not seen in the training set.
Failure Modes: Failures were primarily attributed to segmentation errors (missing parts) or sparse point clouds on thin handles, which biased the Chamfer distance optimization.

5. Significance

This work represents a significant step forward in sample-efficient robotic learning. By shifting from monolithic object representations to compositional part-based modeling, the authors enable robots to:

Generalize skills to objects with drastically different shapes using minimal data.
Handle complex geometric variations (e.g., different handle/spout ratios) that confuse global alignment methods.
Operate effectively in the real world with limited training examples, moving closer to the goal of robots that can learn new tasks from a single human demonstration without extensive retraining.

The approach bridges the gap between high-level semantic understanding (parts) and low-level geometric control (shape warping), offering a robust framework for future adaptive manipulation systems.