🤖 AI

AFFORD2ACT: Affordance-Guided Automatic Keypoint Selection for Generalizable and Lightweight Robotic Manipulation

AFFORD2ACT is a lightweight, affordance-guided framework that distills text and image inputs into a compact set of semantic keypoints, enabling robotic manipulation policies to achieve high generalization and data efficiency across diverse real-world tasks without relying on dense representations or proprioception.

Original authors: Anukriti Singh, Kasra Torshizi, Khuzema Habib, Kelin Yu, Ruohan Gao, Pratap Tokekar

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anukriti Singh, Kasra Torshizi, Khuzema Habib, Kelin Yu, Ruohan Gao, Pratap Tokekar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to use a new tool, like a spatula or a mug. If you show the robot a high-definition video of the whole kitchen, it gets overwhelmed. It sees the shiny countertop, the sunlight reflecting off the window, the pattern on the rug, and the clutter on the table. It tries to learn from all of that, which is like trying to read a book while someone is shouting random words in your ear. It gets confused and fails.

This paper introduces AFFORD2ACT, a clever way to teach robots to ignore the noise and focus only on what matters. Think of it as giving the robot "X-ray vision" for tasks.

Here is how it works, broken down into simple steps:

1. The Problem: Too Much Information

Most robots try to learn by looking at every single pixel in a camera image. This is like trying to learn how to drive a car by memorizing the color of every single leaf on every tree you pass. It's too much data, and the robot gets stuck on details that don't help it move.

2. The Solution: The "Magic Highlighter"

Instead of looking at the whole picture, AFFORD2ACT uses a language prompt (like "Pick up the mug" or "Cut the bread") as a magic highlighter.

Step 1: The Affordance Map. The robot asks a smart AI model, "Where do I need to touch?" The model draws a glowing, fuzzy heat-map over the image, highlighting only the "actionable" parts. For a mug, it highlights the handle. For a knife, it highlights the blade. It ignores the background, the table, and the lighting.
Step 2: Picking the Dots. Once the robot knows where to look, it doesn't look at the whole highlighted area. Instead, it picks just 19 tiny dots (keypoints).
- 15 dots are on the object (e.g., the tip of the handle, the middle of the blade).
- 4 dots are on the robot's own hand (the gripper).
- Analogy: Imagine playing "Connect the Dots." Instead of connecting 10,000 dots to draw a picture, the robot only needs to connect 19 specific dots to know exactly how to move.

3. The "Smart Filter": The Gating System

Here is the really cool part. The importance of those dots changes as the robot moves.

Scenario: Imagine the robot is stirring a pot.
- Phase 1 (Grabbing): The robot needs to focus on the handle of the spoon. The dots on the handle get a "high score," and the dots on the bowl of the spoon get a "low score."
- Phase 2 (Stirring): Now the robot needs to focus on the bowl of the spoon to mix the soup. The robot's "gating system" acts like a dimmer switch, turning up the volume on the bowl dots and turning down the handle dots.

This allows the robot to be lightweight. It doesn't need a super-computer brain to process a whole video; it just needs to track 19 dots and know which ones are important right now.

4. Why This is a Big Deal

The researchers tested this on a real robot arm with six different tasks (pouring, cutting, stirring, kicking a ball, etc.).

It's a Generalist: They trained it on a specific red mug, and it could successfully pick up a blue teapot it had never seen before. Because it learned the shape of the handle (the function), not the color of the mug (the appearance).
It's Robust: Even if there was a person walking by in the background or a messy table, the robot ignored them because they weren't part of the "19 dots."
It's Fast: It learned these skills from very few examples (only 40 tries) and could run in real-time.

The Bottom Line

AFFORD2ACT is like teaching a robot to drive by showing it a map with only the road and the traffic lights drawn on it, instead of showing it a photo of the entire city. By using language to find the "action zones" and then tracking just a few critical points, the robot becomes smarter, faster, and much better at handling new, messy, real-world situations.

1. Problem Statement

Training vision-based robotic manipulation policies that generalize beyond training scenes is challenging due to the "curse of dimensionality" in dense visual inputs (e.g., full RGB images, point clouds). These inputs entangle task-relevant cues with nuisance variations like background clutter, lighting changes, and specific object appearances, often causing policies to overfit to appearance rather than function.

While sparse 2D keypoints offer a compact alternative, automatically selecting task-relevant keypoints that transfer across different objects and instructions is non-trivial. Existing methods suffer from three main limitations:

Motion-driven approaches: Fail in non-static scenes where background motion creates spurious tracks.
Joint learning: Often couples keypoint discovery to specific training data, hindering transfer to new objects.
Semantic/Object-centric approaches: Identify where an object is but fail to identify where to interact (e.g., distinguishing between grasping a brush handle vs. touching the bristles for sweeping).

The core challenge is to automatically extract functionally grounded, task-conditioned keypoints that generalize across object shapes while remaining robust to distractors.

2. Methodology: AFFORD2ACT

The proposed framework, AFFORD2ACT, uses language-conditioned affordance priors to localize interaction regions and extract a small set of semantic 2D keypoints. The pipeline consists of four main stages (illustrated in Fig. 2 of the paper):

A. Affordance-Guided Keypoint Distillation

Affordance Localization: Given a single-view RGB frame and a natural language prompt (e.g., "hold the brush"), a pre-trained One-Shot Open Affordance model generates a heatmap indicating actionable regions. A robust quantile threshold converts this into a binary mask ( $M$ ), filtering out clutter and focusing only on the interaction surface.
Reference Keypoint Extraction: On a reference frame, dense feature maps are generated using a DINO backbone (self-supervised vision transformer). Features within the affordance mask are clustered to select $K$ reference anchor points (e.g., 15 points on the object, 4 on the robot end-effector).
Cross-instance Correspondence: For new object instances, the system matches the reference anchors to the new frame by computing cosine similarity between DINO features, restricted to the affordance mask. This ensures that index $i$ always refers to the same functional part (e.g., "blade tip") regardless of geometry or appearance changes.
Temporal Tracking: A transformer-based point tracker (e.g., CoTracker) tracks these $K$ keypoints over time, ensuring index consistency even during occlusions (e.g., when the gripper blocks the view).

B. Policy Design with Gating

The policy learns to act based on the sparse trajectory of these keypoints.

Tokenization: Keypoints are embedded into tokens and processed by a Transformer Encoder to contextualize them (e.g., understanding the relationship between a knife blade and its handle).
Gating Mechanism: A lightweight gating network assigns dynamic importance weights to each keypoint token at every timestep. This allows the policy to prioritize task-relevant keypoints and ignore irrelevant ones as the interaction phase changes (e.g., focusing on the handle during grasping, then the tip during cutting).
Action Output: The weighted sum of tokens forms a compact scene embedding, which is passed through an MLP to predict end-effector deltas and gripper commands.

3. Key Contributions

Automatic, Language-Conditioned Keypoint Extraction: A fully automatic pipeline that selects sparse 2D keypoints from interaction-relevant regions without manual annotation, guided by affordance models.
Correspondence-and-Gating Policy: A novel formulation that preserves semantic consistency across object instances while dynamically adapting keypoint relevance via a gating network during execution.
Robust Real-Robot Generalization: Extensive evaluation on six real-world tasks demonstrating robust transfer to held-out objects, novel categories, cluttered scenes, and open-vocabulary prompt variations.

4. Experimental Results

The method was evaluated on six tasks: grasp-and-lift, cutting, sweeping, pouring, stirring, and ball-kicking, using a UR3e robot arm.

Performance: AFFORD2ACT achieved an 82% success rate on held-out objects and 86% on seen objects.
Comparison with Keypoint Priors (E1): It significantly outperformed baselines using YOLO+DINO, YOLO+Uniform grids, and SAM+DINO. It matched the performance of manual expert selection (P3PO-style) without requiring per-task point specification.
Comparison with Dense Inputs (E2): The keypoint-based policy outperformed policies trained on raw RGB images, RGB-D, and dense affordance masks (UAD), particularly on unseen objects. Dense inputs tended to overfit to specific environments.
Data Efficiency: The method remained highly effective even with only 10 demonstrations, whereas image-based baselines degraded significantly as data was reduced.
Generalization (E3): The policy successfully transferred to:
- Novel Instances: Different mugs or brushes within the same category.
- Novel Categories: Objects with shared functional parts (e.g., stirring with a spatula instead of a spoon).
- Distractors: Robustness against background changes and moving agents (e.g., a person nearby).
Ablation Studies: Removing the gating network caused a drastic drop in success rate (from 86% to 52%), proving the necessity of dynamically weighting keypoints. The effective number of active keypoints ( $K_{eff}$ ) was found to be much smaller than the total set ( $K=19$ ) at any given time.

5. Significance and Future Work

Significance:
AFFORD2ACT demonstrates that affordance-guided sparse representations are a superior alternative to dense visual inputs for robotic manipulation. By decoupling "what to see" (affordance) from "how to act" (policy), the method achieves:

Lightweight computation: Enabling real-time inference with fewer parameters.
High generalizability: Reducing reliance on specific object classes or training environments.
Data efficiency: Learning complex skills from very few demonstrations.

Limitations & Future Directions:

Failure Modes: Performance degrades if the affordance mask is inaccurate (e.g., missing the true contact area) or if occlusion is severe in a single-view setup.
Future Work: The authors propose integrating online keypoint resampling and lightweight task planning to handle long-horizon, multi-phase tasks (e.g., grasping then stirring) and improving robustness to partial observability.

In conclusion, AFFORD2ACT provides a scalable, efficient, and generalizable framework for robotic manipulation by leveraging language-conditioned affordances to distill visual complexity into actionable semantic keypoints.