🤖 AI

AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence

AffordGen leverages 3D generative models and vision foundation models to create diverse, affordance-aware demonstration datasets via semantic keypoint correspondence, enabling robot manipulation policies to achieve robust zero-shot generalization to unseen objects with improved data efficiency.

Original authors: Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiawei Zhang, Kaizhe Hu, Yingqian Huang, Yuanchen Ju, Zhengrong Xue, Huazhe Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to pour tea. In the old days, you would have to physically hold the robot's hand and guide it through the motion hundreds of times with different teapots, cups, and angles. If you showed it a teapot with a handle on the left, it might get confused when faced with a teapot with a handle on the right. It's like teaching a child to ride a bike by only letting them practice on a single, specific bike; if you swap it for a different model, they might fall over.

This is the problem AffordGen solves. It's a new system that lets a robot learn from just one demonstration and then instantly figure out how to do the same task with thousands of different objects it has never seen before.

Here is how it works, broken down with some everyday analogies:

1. The "Magic Translator" (Affordance Correspondence)

Usually, robots see objects as just a pile of 3D dots (a point cloud). They don't really understand what the object is or how to use it.

AffordGen uses a "Magic Translator" (powered by advanced AI vision models). Instead of just looking at the shape, it looks for functional landmarks.

The Analogy: Imagine you are teaching someone how to use a strange new tool. You don't say, "Grab the red metal part." You say, "Grab the handle."
How it works: The system identifies the "handle" (where you grab) and the "spout" (where the tea comes out) on the original teapot. It then uses its "Magic Translator" to find the exact same "handle" and "spout" on a completely different object, like a weirdly shaped mug or even a teapot made of glass. It understands that functionally, these two different shapes are the same.

2. The "Shape-Shifting Copycat" (Generative Data)

Once the robot knows where to grab and where to pour on the new object, it needs to practice. But we don't want to spend hours filming the robot again.

AffordGen acts like a generative copycat.

The Analogy: Imagine you have a video of a dancer doing a specific routine. Instead of just playing that video back, you use a computer program to instantly re-enact that same dance routine on a giant, a tiny person, and a person with a different body shape, all while keeping the dance steps perfect.
How it works: The system takes the one video you gave it, finds the "handle" and "spout" on a new 3D model, and mathematically "morphs" the robot's movement to fit that new shape. It does this thousands of times, creating a massive library of practice videos for objects the robot has never actually touched.

3. The "Reactive Athlete" vs. The "Scripted Actor"

There are two ways to teach a robot:

The Scripted Actor (Old Way): You give the robot a pre-written script: "Move arm 5 inches left, then 2 inches down." If the object moves slightly, the robot misses because it's just following the script blindly.
The Reactive Athlete (AffordGen Way): AffordGen uses the thousands of generated practice videos to train the robot to be an athlete. It learns the feel of the task. If the cup is slightly tilted or the handle is in a weird spot, the robot reacts in real-time, just like a human would, because it has "seen" so many variations of the task during its training.

Why is this a Big Deal?

Efficiency: Instead of needing 1,000 hours of human demonstration, you might only need one. The system generates the other 999 hours of practice data automatically.
Generalization: It works on things the robot has never seen. If you teach it to pour tea from a ceramic pot, it can figure out how to pour from a plastic bottle or a metal jug, because it understands the concept of "pouring," not just the specific shape of the pot.
Real-World Ready: The paper shows this working not just in computer simulations, but on real robots in the real world, successfully handling objects it has never encountered before.

In a Nutshell

AffordGen is like giving a robot a "universal manual" for how to interact with the world. Instead of memorizing every single object in existence, it learns the rules of interaction (like "grab the handle," "pour from the spout"). Once it knows the rules, it can apply them to any new object it encounters, making robots much more adaptable and useful in our messy, diverse real world.

1. Problem Statement

Robot manipulation via visuomotor imitation learning faces two critical bottlenecks:

Data Scarcity: Collecting large-scale, high-quality human demonstrations for diverse objects and scenarios is prohibitively expensive and time-consuming.
Poor Generalization: Existing methods struggle to generalize to novel objects (different shapes, sizes, or even categories) not seen during training.
- Current Synthetic Methods: Approaches like DemoGen and CPGen augment data by spatially transforming a single object instance. While they improve spatial diversity, they fail to generalize across different object instances or categories because they lack semantic understanding.
- Current Affordance Methods: Approaches like Robo-ABC or FUNCTO use semantic correspondence (affordances) to transfer skills but rely on open-loop planning. They lack the reactive robustness of closed-loop, end-to-end learned policies.

Goal: To create a framework that scales a minimal number of human demonstrations into thousands of diverse, semantically grounded trajectories across novel object instances and categories, enabling robust, zero-shot generalization.

2. Methodology: AffordGen

AffordGen is a novel framework that repurposes affordance correspondence as a generative prior. Instead of using affordances for online planning, it uses them to synthesize training data for learning a closed-loop visuomotor policy.

The pipeline consists of three main stages:

A. Source Demonstration Pre-processing

Given a single expert demonstration, the system extracts:

Grasp Time ( $t_{grasp}$ ): When the gripper closes.
Skill Segment ( $\tau_s$ ): The critical trajectory for task execution (e.g., pouring, cutting).
Key Points:
- Affording Point: The contact point between the gripper and the object.
- Function Point: The point on the tool/object that interacts with the goal (e.g., the spout of a teapot).
Data Processing: The visual input is processed using RGB-D cameras. Objects are segmented (using SAM2) and converted into point clouds, which are down-sampled using Farthest Point Sampling (FPS) to create the workspace point cloud.

B. Semantic Correspondence on 3D Meshes

To transfer trajectories to new objects, AffordGen establishes 3D correspondences between the source object and a library of unseen 3D meshes:

Canonical Normalization: All 3D meshes are normalized into a unified canonical space using a 6D pose estimator.
Multi-view Rendering: Parallel rendering generates RGB-D images from multiple camera angles.
Feature Matching: A Vision Foundation Model (DINOv2) extracts semantic features from the rendered images.
Correspondence Calculation: The system finds the nearest neighbors in the feature space between the source mesh vertices and the target mesh vertices. The target keypoint is calculated via a weighted average of candidate correspondences based on cosine similarity scores.

C. Diverse Demonstration Generation

Once correspondences are established, the system generates new trajectories for novel objects:

Keypoint-Constrained Replay:
- Grasp Segment ( $\tau_g$ ): Transferred by aligning the relative trajectory of the end-effector to the affording point of the new object.
- Skill Segment ( $\tau_s$ ): Transferred by aligning the relative trajectory of the function point to the new object's function point.
- This assumes that for objects of the same functional class, the relative motion between the robot and these semantic points remains consistent.
Transition Planning: The segments connecting grasp and skill are generated using collision-free motion planning (e.g., RRT*) or spherical linear interpolation.
Point Cloud Synthesis: To create realistic training data, the system renders the robot and manipulated object point clouds from simulation and replaces the corresponding parts in the source data. This creates a "hybrid" real-simulated dataset that mitigates the sim-to-real gap without full environment reconstruction.

3. Key Contributions

AffordGen Framework: A novel approach that transforms static affordance mappings into a generative source for data synthesis, bridging the gap between semantic correspondence and end-to-end learning.
Scalable Data Generation: The ability to scale a single human demonstration into thousands of diverse trajectories covering novel object instances, full 6D poses, and even cross-category objects (e.g., transferring "pouring" from a teapot to a mug).
Robust Closed-Loop Policies: By training on this massive, semantically consistent dataset, the resulting policies inherit the generalizability of affordances and the reactivity of learning-based control, overcoming the limitations of open-loop planners.
Zero-Shot Generalization: Demonstrated capability to generalize to truly unseen objects and categories without additional fine-tuning.

4. Experimental Results

Simulation Experiments

Tasks: Teapot pouring, Mug hanging, Knife cutting, Shoe organizing.
Baselines: Compared against DemoGen (spatial augmentation) and CPGen (shape stretching).
In-Category Performance: AffordGen achieved an average 24.1% performance boost over the best baseline on unseen objects within the same category.
Cross-Category Performance: In zero-shot tasks (e.g., Mug Pouring learned from Teapot Pouring), AffordGen was the only method to achieve meaningful non-zero success rates, whereas baselines failed completely.

Real-World Experiments

Setup: Trained solely on generated data from 10 real-world expert demonstrations. Evaluated on unseen real objects.
Baselines: Included DemoGen, CPGen, and FUNCTO (a state-of-the-art affordance-based planner).
Results:
- AffordGen significantly outperformed all baselines, including the planner-based FUNCTO.
- FUNCTO Limitations: FUNTO struggled with occlusion and large orientation changes, as it relies on precise keypoint detection for open-loop planning.
- AffordGen Strength: By learning the relationship between affordances and object shapes from large-scale data, AffordGen implicitly handles occlusion and orientation variance, achieving high success rates (e.g., 74/162 success on unseen Mug Hanging vs. 50/162 for FUNTO).

5. Significance

AffordGen represents a paradigm shift in robot learning by decoupling the semantic knowledge of affordances from the execution mechanism of planning.

Data Efficiency: It drastically reduces the need for human data collection, enabling robots to learn complex manipulation skills from very few demonstrations.
Generalization: It solves the "long-tail" problem in robotics, allowing robots to interact with objects they have never seen before, provided the objects share functional affordances.
Practicality: The method is applicable to real-world deployment, offering a scalable path toward general-purpose robotic assistants capable of handling diverse household and industrial tasks.