💻 computer science

GRAFT: Graph-Based Affordance Transfer via Part Correspondence

GRAFT is a geometry-aware framework that enables zero-shot robotic manipulation transfer by representing objects as part-based graphs to retrieve functionally and geometrically similar instances and propagate contact points through fine-grained part correspondence.

Original authors: Mengying Lin, Utkarsh Mishra, Ajay Mandlekar, Danfei Xu

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mengying Lin, Utkarsh Mishra, Ajay Mandlekar, Danfei Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to pick up a new, strange object it has never seen before, like a weirdly shaped watering can. The old way of doing this was like asking a librarian to find a book based only on the title. If you asked for a "mug," the librarian would only show you other mugs. If you asked for a "watering can," they'd only show you watering cans. But what if the handle on the watering can is shaped exactly like the handle on a mug? The old method would miss that connection because the titles (the object names) are different, even though the parts are the same.

This paper introduces GRAFT, a new way to teach robots that looks at the parts of an object rather than just the whole thing. Think of it like a master carpenter who doesn't just look at a chair; they look at the legs, the seat, and the backrest to understand how to build or fix it.

Here is how GRAFT works, broken down into simple steps:

1. Turning Objects into "Family Trees" (Graphs)

Instead of looking at an object as one big blob, GRAFT breaks it down into a map of its parts.

The Nodes: Imagine every handle, spout, or base is a node on a family tree.
The Connections: The lines connecting them show how those parts are arranged in space (e.g., the handle is attached to the side of the cup).
The "DNA": Each part has a description of its shape and what humans usually do with it (like "this is where you hold it").

2. The "Matchmaker" Algorithm (UFGW)

Now, the robot has a new object (the "Target") and a library of old demonstrations (the "Source"). It needs to find the best match.

The Old Way: It would try to match the whole object. "Is this watering can like a teapot?" No. "Is it like a bottle?" No.
The GRAFT Way: It uses a special math tool called UFGW (Unbalanced Fused Gromov–Wasserstein). Think of this as a super-smart matchmaker that says, "I don't care if the whole object looks different. I see that this watering can has a handle that looks and acts exactly like the handle on this teapot."
The "Unbalanced" Trick: Sometimes objects have different numbers of parts (a mug has a handle, a bowl doesn't). The "unbalanced" part of the math allows the robot to say, "Okay, I'll match the handle to the handle, and I'll just ignore the extra parts that don't have a match." This makes the matching much more flexible.

3. Finding the "Root" (Where to Hold)

To make sure the robot knows where to grab, GRAFT looks at the demonstrations to see which part was touched first (the "root").

It uses a process called EM Optimization (think of it as a "trial and error" loop).
The Problem: If the robot just guesses the best match, it might grab the wrong part because two parts look similar locally (like grabbing the top of a bottle instead of the neck).
The Solution: The EM process looks at the whole family tree structure to decide which part is the most important "root." It ensures the robot grabs the functional part (like a handle) rather than just a visually similar but useless part.

4. The Result: Zero-Shot Magic

Once the robot finds the matching parts, it "transfers" the knowledge.

If a human demonstrated how to grab a teapot by its handle, GRAFT finds the watering can's handle and says, "Grab here!"
It then tells the robot's hand exactly where to move.
The "Zero-Shot" part: The robot never saw that specific watering can before. It didn't need to be trained on it. It just used the logic of parts to figure it out instantly.

Why is this better?

The paper tested this in simulations and with real robots.

More Variety: Because it matches parts, it can use a demonstration from a teapot to teach a robot how to hold a mug, a bottle, or a weirdly shaped tool. Old methods were too picky and only worked on nearly identical objects.
Better Success: In tests, GRAFT successfully grabbed new objects about 81% of the time, while other methods struggled around 36-43%.
Data Generator: The authors also showed that GRAFT can be used to automatically create thousands of new training examples for robots by taking a few real demonstrations and "grafting" them onto new shapes, making it easier to train robots for the real world.

In short: GRAFT stops robots from being obsessed with the "name" of an object and starts them thinking about the "anatomy" of an object. It's like teaching a robot to recognize that a door handle and a faucet handle are cousins, even if one opens a door and the other turns on water.

Technical Summary: GRAFT: Graph-Based Affordance Transfer via Part Correspondence

Problem Statement

Generalizing robotic manipulation to unseen objects remains a significant challenge for learning-based approaches, which often require extensive demonstrations and struggle in few-shot or zero-shot settings. Prior retrieval-based methods attempt to transfer manipulation affordances (e.g., grasps) by finding similar source objects in a dataset. However, existing pipelines typically rely on high-level semantic filtering (e.g., category names) followed by geometric alignment using appearance-based descriptors (e.g., from foundation models like DINOv2). The authors argue that this semantic-to-geometric workflow is overly conservative; it severely limits the diversity of retrievable objects because it demands that source and target objects look alike globally. Consequently, these methods fail to identify functionally compatible objects that belong to different semantic categories but share similar part-level geometries (e.g., a mug handle vs. a watering can handle).

Methodology

The paper proposes GRAFT (Graph-Based Affordance Transfer), a learning-free, geometry-aware correspondence framework designed to enable zero-shot manipulation transfer using only one demonstration per object. The core insight is that functional compatibility is determined by an object's structural organization rather than its holistic appearance.

1. Part-Based Graph Representation

Each object is represented as an undirected graph where:

Nodes correspond to individual object parts. Each node encodes multi-level descriptors:
- Geometric: Oriented Bounding Box (OBB) extents and normalized part point clouds.
- Vertex-level: Canonical positional encodings (Fourier-style) to enable fine-grained point-wise matching within parts.
- Global/Functional: A descriptor extracted from a foundation model (e.g., DINOv2) and a binary annotation indicating if the part is an interactable region (based on demonstrations).
Edges capture the spatial relationships between parts, storing relative translation and orientation to preserve the object's assembly structure.

2. Unbalanced Fused Gromov–Wasserstein (UFGW) Distance

To compare these graphs, GRAFT employs the UFGW distance. Unlike standard graph matching, UFGW:

Jointly optimizes for node feature similarity and graph-level structural consistency.
Supports partial matching between graphs of unequal size or structure (e.g., objects with different numbers of parts).
Allows for mass variation, which is crucial for handling objects with different scales or articulations.

3. Node Mass Estimation and Optimization

A critical component of the framework is the assignment of "mass" to nodes, representing their structural and functional importance:

Source Nodes: Mass is fixed based on the OBB volume and a saliency factor that decays exponentially with the Breadth-First Search (BFS) distance from the demonstrated contact part (the root node). This ensures the interactable region and its immediate neighbors carry higher weight.
Target Nodes: Since target objects lack interaction annotations, their node masses are not predefined. GRAFT uses an Expectation-Maximization (EM) procedure integrated into the UFGW optimization:
- E-step: Solves for the optimal transport plan $T$ given current target masses.
- M-step: Updates target masses by combining a geometric prior (normalized OBB volumes) with the empirical flow from the transport plan. This adaptive weighting improves robustness against variations in part scale and relevance.

4. Retrieval and Transfer Pipeline

Retrieval: For a target object, the system computes the UFGW distance against all source objects in the demonstration buffer. The top- $K$ most functionally similar objects are selected.
Alignment: The optimal transport plan $T$ provides a mapping between source and target parts.
Contact Propagation: Contact points from the source demonstration are propagated to the target object through the established part-level correspondences.
Execution: The transferred contact points are fed into a grasp planner (e.g., AnyGrasp) and a motion planner to generate executable trajectories.

Key Contributions

Structure-Driven Retrieval: The paper introduces a paradigm shift from object-level semantic/geometric similarity to part-level structural correspondence. This allows the system to retrieve functionally similar objects even if they belong to disjoint semantic categories.
Learning-Free Framework: GRAFT operates without training a new model for specific tasks, relying instead on geometric descriptors and the UFGW metric.
EM-Based Mass Optimization: The novel integration of an EM procedure to infer target node masses enables robust matching even when the target object's functional parts are unknown.
Scalable Data Generation: By integrating with MimicGen, GRAFT serves as a structural prior for synthesizing diverse, physically valid manipulation demonstrations for unseen objects from a small set of seed demonstrations.

Experimental Results

The authors evaluate GRAFT across three settings:

Zero-Shot Affordance Generalization:
- Compared against baselines like where2act and ROBO-ABC (using CLIP or DINO features), GRAFT achieved superior performance.
- Metrics: GRAFT attained an Affordance Success Rate (ASR) of 0.85 (vs. 0.69 for the best baseline) and a Normalized Semantic Similarity (NSS) of 0.82, while significantly reducing the Distance-to-Motion (DTM) to 0.02.
Simulation-Based Execution:
- In physics-based simulations (SAPIEN) with a floating gripper, GRAFT achieved a Simulation Success Rate (SR) of 81.25%, significantly outperforming baselines (e.g., 43.75% for ROBO-ABC with DINO).
- Crucially, GRAFT achieved 0% AnyGrasp Error, indicating that the transferred grasp points were consistently valid, whereas baselines frequently failed to predict a valid grasp pose near the target.
Data Generation (MimicGen Integration):
- When used to generate training trajectories for unseen objects, GRAFT achieved a 63% trajectory success rate in Mujoco, compared to only 11% for vanilla MimicGen (which lacks correspondence reasoning).
Real-World Validation:
- The system was tested on a Franka Panda robot using real-world scanned objects. It successfully transferred grasps from source objects to unseen targets (e.g., mugs, briefcases) and executed stable lifts.

Ablation Studies

The authors conducted ablation studies to validate the necessity of each component:

Removing Graph Structure: Caused the largest performance drop (SR dropped to 0.38), confirming that part-level structural relations are critical.
Replacing DINO with CLIP: Resulted in a slight decline in NSS, suggesting DINO's self-supervised features better capture geometric correspondences.
Removing BFS Saliency: Reduced performance, highlighting the importance of weighting parts based on their proximity to the interaction point.
Greedy Root Selection (No EM): Using a greedy approach to select the target root node led to suboptimal alignment (SR 0.77 vs. 0.85), demonstrating that EM optimization is necessary to resolve global structural consistency.

Significance and Claims

The paper claims that GRAFT establishes a robust foundation for correspondence-driven reasoning in robotic manipulation. By moving away from conservative, appearance-based retrieval toward structure-driven part correspondence, the method enables:

Greater Diversity: Retrieving a wider range of functionally similar objects, including those from different semantic categories.
Zero-Shot Generalization: Successfully transferring manipulation skills to entirely new object instances without fine-tuning.
Scalable Data Synthesis: Generating high-quality, physically valid training data for unseen objects, which is essential for training policies that generalize in open-world settings.

The authors acknowledge limitations, noting that the method assumes rigid bodies (deformable objects may violate structural assumptions) and is currently evaluated primarily on pick-and-place tasks. However, they position GRAFT as a significant step toward data-efficient, open-world manipulation.