💻 computer science

AFUN: Towards an Affordance Foundation Model for Functionality Understanding

This paper introduces AFUN, an affordance foundation model that predicts task-conditional functional masks and 3D post-contact motion curves from RGB-D observations and language descriptions, achieving state-of-the-art performance in segmentation, contact-point prediction, and motion planning while enabling zero-shot deployment for real-world robot manipulation across diverse open-world environments.

Original authors: Zhaoning Wang, Yi Zhong, Jiawei Fu, Henrik I. Christensen, Jun Gao

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaoning Wang, Yi Zhong, Jiawei Fu, Henrik I. Christensen, Jun Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a brand-new kitchen. You don't need a manual to know that the handle on the cabinet is for pulling, the knob on the stove is for turning, and the lid on the pot is for lifting. You instantly understand not just what an object is, but how to use it. In the world of robotics, this intuitive understanding is called "affordance."

For a long time, robots have struggled with this. They might know a drawer exists, but they don't know exactly where to grab it or how to pull it open without breaking it. Existing AI models were like students who could only answer half the question: some could point to the handle but didn't know how to pull it, while others could guess the motion but didn't know which object to touch.

Enter AFUN (Affordance Foundation Model for Functionality Understanding). Think of AFUN as a robot's "super-intuitive sense" that combines vision, language, and physical movement into one package.

Here is how AFUN works, broken down into simple parts:

1. The "Big Library" of Experience

To learn how to use objects, you need to see many people and robots doing it. The researchers built a massive "library" of data by gathering videos from everywhere:

Real robots doing tasks.
Humans filming themselves from a first-person view (like GoPro footage).
Simulations in video games.
3D scans of real rooms.

They took all this messy, different data and cleaned it up into a single, standard format. It's like taking recipes from French, Italian, and Chinese chefs and translating them all into one universal cookbook so the robot can learn from everyone at once.

2. The Two-Step "Magic Trick"

When you give AFUN a picture of a room and a command like "Open the microwave," it doesn't just guess. It performs two specific tasks simultaneously:

Step A: The "Where" (The Mask): It draws a digital highlighter over the exact part of the microwave you need to touch (the handle or the door). It ignores the buttons or the top of the machine.
Step B: The "How" (The 3D Curve): It doesn't just stop at the handle. It draws a smooth, 3D line in the air showing exactly how the door should move. Is it a straight pull? A twist? A lift? AFUN predicts this path as a smooth curve, like a rollercoaster track the robot's hand can follow.

3. How It Learns (The "Teacher" and the "Student")

The model uses a clever trick to learn. It uses a giant, pre-trained "brain" (a Vision-Language Model) that already knows how to understand pictures and words.

The Teacher: This big brain reads the instruction ("Open the microwave") and looks at the picture.
The Student: AFUN uses special "query tokens" (think of them as sticky notes) to ask the big brain: "Show me the handle!" and "Show me the motion!"
The Result: The big brain guides AFUN to draw the mask and the 3D curve.

4. Real-World Testing

The researchers didn't just test this on a computer; they put it on a real robot arm (a Franka robot).

The Test: They asked the robot to do things like "Pick up the screwdriver," "Take off the pot lid," and "Open the microwave."
The Outcome: The robot successfully figured out where to grab and how to move the object without needing any special programming for that specific robot or task. It just looked, listened, and acted.

Why This Matters (According to the Paper)

The paper claims that AFUN is a major step forward because:

It's General: It works on objects and tasks it has never seen before, not just the ones it was memorized on.
It's Complete: It solves both the "where to touch" and "how to move" problems at the same time.
It's Ready: It can be deployed on real robots immediately, without needing to be re-taught for every new machine.

In short, AFUN gives robots the ability to look at a new object, understand a human's request, and physically figure out the best way to interact with it, just like a human would.

Technical Summary: AFUN: Towards an Affordance Foundation Model for Functionality Understanding

Problem Statement
Affordance understanding serves as a critical interface between visual perception and physical action, enabling robots to operate in open, unstructured environments. However, constructing a foundation model that simultaneously generalizes across diverse environments, objects, and tasks remains a significant challenge. Existing methods typically address only partial aspects of this problem: some localize task-relevant regions (where to interact) without specifying executable motion, while others predict motion but lack scalability or require heuristic localization. Furthermore, current approaches often fail to characterize how an object should move after interaction (e.g., the specific trajectory for opening a drawer vs. lifting a lid), and most deep learning models suffer from limited open-world generalization due to small-scale, non-diverse datasets.

Methodology
The authors present AFUN, a unified foundation model designed to predict both where to interact and how to interact from a single RGB-D observation and a natural language task description.

Unified Data Pipeline: To support open-world generalization, the authors constructed a large-scale, standardized data pipeline. They aggregated 321,190 raw videos from 10 public sources, including robot teleoperation, human egocentric videos, simulations, and real-world scans. The pipeline converts these heterogeneous sources into a shared schema containing:
- Task Description: Natural language instructions.
- Functional Mask: A segmentation mask identifying the manipulable part.
- 3D Motion: A post-contact 3D trajectory represented as a Bézier spline curve.
- Key Innovation: Unlike prior works that use hand/gripper trajectories (which include pre-contact noise), AFUN tracks the object's motion itself. It utilizes a Vision-Language Model (Qwen3-VL) to generate part-level queries and SAM3 to track the manipulated object, subsequently back-projecting the mask to recover a smooth 3D motion curve.
Model Architecture: AFUN employs a dual-branch architecture conditioned on a frozen Vision-Language Model (Qwen3-VL-8B) and a frozen segmentation model (SAM3).
- MetaQuery Conditioning: The model uses learnable "MetaQuery" tokens to bridge the VLM with downstream tasks. One set of tokens ($mqs$) connects to the segmentation model, and another ($mqm$) connects to the motion decoder. This allows the model to share reasoning capabilities from the VLM in a single forward pass.
- Segmentation Branch: Uses SAM3 to predict task-conditioned functional masks based on the semantic tokens.
- Motion Branch: Uses a 3D feature encoder (Sonata) to extract geometric features from the depth input. A transformer decoder predicts the parameters of a 3D Bézier spline curve. The curve is anchored at the contact centroid ( $P_0$ ) and defined by predicted control points in relative 3D coordinates.
Training Scheme: The model is trained in three stages to ensure stability:
1. Alignment: Aligning MetaQuery tokens with SAM3's text-conditioning space using the Visual Genome dataset.
2. Segmentation: End-to-end training of the segmentation branch on aggregated affordance datasets (HOVA-500K, RAGNet, InstructPart, ReasonAFF).
3. Joint Training: Jointly training segmentation and motion prediction on the curated AFUN dataset, using a combined loss function of segmentation grounding and curve fitting (point-sampling loss).

Key Contributions

AFUN Model: A foundation model that jointly predicts task-conditional functional masks and 3D post-contact motion curves, bridging the gap between "where" and "how" in robot manipulation.
Large-Scale Standardized Dataset: The creation of one of the largest public affordance datasets to date, featuring 59,867 high-quality training samples with language, masks, and object-centric 3D motion labels derived from heterogeneous sources.
Object-Centric Motion Tracking: A novel data processing strategy that tracks the object's motion rather than the gripper's, avoiding pre-contact noise and providing a direct representation of functional interaction.
Zero-Shot Deployment: The ability to deploy the model on real robots without task-specific heuristics or embodiment-specific finetuning.

Experimental Results
AFUN was evaluated on 8 segmentation benchmarks and 3 motion benchmarks, demonstrating superior performance across all metrics:

Affordance Segmentation: AFUN achieved a mean gIoU of 69.3% and cIoU of 67.2%, outperforming the strongest baseline (Affordance-R1) by +23.9/+26.3 points. It also outperformed baselines on complex, intent-dependent regions (e.g., distinguishing a shovel blade for "containing" vs. a handle for "using").
Contact Point Prediction: Using the Pole of Inaccessibility of the predicted mask, AFUN achieved a hit rate gain of 12.7%–61.3% over the best baselines (e.g., 96.5% on InstructPart vs. 35.2% for VRB).
3D Motion Prediction: AFUN achieved the lowest Average Displacement Error (ADE) and Final Displacement Error (FDE) on all three test sets (AFUN test set, SceneFun3D, and RoboMIND2). It significantly outperformed baselines in Contact-in-Mask (CIM) hit rates (e.g., 81.0% on the AFUN test set vs. 11.0% for VRB).
Real-Robot Deployment: The model was deployed on a Franka Research 3 arm for tasks like opening a microwave, taking off a pot lid, and opening a drawer. It achieved an average success rate of 90% without any robot-specific finetuning.

Significance and Claims
The paper claims that AFUN represents a significant step toward an open-world affordance foundation model. By unifying functionality perception (mask) with executable action (3D motion) and leveraging a massive, standardized dataset, AFUN demonstrates the ability to adapt to open-world tasks without relying on narrow domain data or manual heuristics. The authors emphasize that the model's ability to predict smooth, orientation-aware 3D trajectories (via Bézier splines) allows for complex manipulations (like rotational opening) that are difficult for prior line-based or discrete motion approaches. The work suggests a practical path for robots to understand and execute diverse physical interactions in unstructured environments.