🤖 AI

Token-Based Affordance Grounding with Large Vision-Language Models

This paper proposes TokAG, a zero-shot affordance grounding framework that leverages a spatial-aware token-selection mechanism to extract object-focused attention maps from Large Vision-Language Models, thereby achieving superior performance over prior weakly supervised methods in localizing action-relevant regions without external supervision.

Original authors: Seung Il Lee, Qinqian Lei, Daguang Xu, Dong Yang, Robby T. Tan, Yixin Chen, Bo Wang

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seung Il Lee, Qinqian Lei, Daguang Xu, Dong Yang, Robby T. Tan, Yixin Chen, Bo Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to interact with the world. You don't just want the robot to know what an object is (e.g., "That is a chair"). You want it to know how to use it (e.g., "Sit here," "Push here," or "Cut with this"). This ability to understand where an action happens is called Affordance Grounding.

The paper introduces a new method called TokAG that helps computers figure out exactly where on an object a human should perform an action, without needing to be manually taught with thousands of labeled examples.

Here is how it works, explained through simple analogies:

The Problem: The "Blind" Guide

Previous methods were like a student trying to guess where to sit on a motorcycle by looking at a blurry photo and a very short note that just says "Sit."

The Confusion: If the photo shows a motorcycle and the note says "Sit," the computer might get confused. Is it the seat? The handlebars? The gas tank?
The Limitation: Old methods often relied on short, vague instructions. They struggled when an image had multiple things happening at once (like someone holding a toothbrush and brushing with it) or when two actions looked similar (like "pushing" a bike vs. "sitting" on it). They often pointed to the whole object instead of the specific part needed for the action.

The Solution: The "Super-Reader" (LVLM)

The authors realized that Large Vision-Language Models (LVLMs)—the same type of AI that can look at a picture and write a long, detailed story about it—are actually very smart about where things are, even if they don't say it out loud.

Think of an LVLM as a Super-Reader who looks at a picture and a question (e.g., "Which part of the toothbrush is used for brushing?").

The Super-Reader doesn't just spit out the answer "bristles."
While it is "thinking" and generating that answer word-by-word, it is also secretly pointing its internal "finger" (attention) at different parts of the image.
Sometimes, it points at the background (the bathroom sink). Sometimes, it points at the whole toothbrush. But when it thinks of the word "bristles," its internal finger points very strongly at the bristles.

The Innovation: The "Spotlight Selector" (TokAG)

The tricky part is that the Super-Reader generates a whole sentence, and it points its finger at many different things for every single word it says. If you just look at the whole sentence, the "pointing" gets messy and blurry.

TokAG is like a Spotlight Selector that solves this mess:

The Question: It asks the Super-Reader a specific question about an action (e.g., "When people sit on a bed, which part is used?").
The Word-by-Word Scan: As the AI generates the answer (e.g., "The mattress"), TokAG looks at the "attention map" (the internal pointing) for every single word it produces.
The Filter: It ignores words that point at the background or the whole room. It looks for the specific word where the AI's internal pointing is most concentrated on the object.
- Example: When the AI says "mattress," its attention is tightly focused on the bed's surface. When it says "bed," the attention might be spread out. TokAG picks the "mattress" moment.
The Result: It takes that specific "moment of focus" and turns it into a clear heat map, showing exactly where to sit.

Why It's Better

No Training Required: Unlike previous methods that needed to be "trained" on millions of labeled photos (like a student memorizing a textbook), TokAG works zero-shot. It just uses the knowledge the AI already has, like a smart person figuring out a new situation without needing a manual.
Precision: It can tell the difference between "sitting on" a motorcycle (the seat) and "pushing" a motorcycle (the handlebars), even though the object is the same.
Performance: The paper shows that TokAG is significantly better than previous methods. On standard tests, it improved accuracy by about 10% to 30% compared to the best existing techniques, even though it didn't use any training data.

The Catch (Limitations)

The paper notes that this method relies on the AI generating a single "key word" that points to the right spot.

Complex Actions: If an action requires two hands or two different parts at once (like "opening a jar" which needs a hand on the lid and a hand on the jar), the current method might struggle because it's looking for one single "best" word to point to.
Distractions: If an object has a bright logo or text on it, the AI might get distracted and point at the logo instead of the functional part.

Summary

TokAG is a clever trick that turns a "chatbot" AI into a precise "pointer." Instead of asking the AI to draw a box or write coordinates, it asks the AI to describe the action, then listens to which word causes the AI to look most intently at the right part of the object. It turns the AI's internal "thinking process" into a precise map for robots and computers to follow.

Technical Summary: Token-Based Affordance Grounding with Large Vision-Language Models (TokAG)

Problem Statement
Affordance grounding aims to localize specific image regions that support potential human actions, a capability essential for physical intelligence and embodied perception. While recent advances in weakly supervised learning have utilized exocentric images with action-level supervision, these methods face two primary limitations:

Visual Ambiguity in Co-occurring Actions: Exocentric images often contain multiple simultaneous actions (e.g., "holding" and "brushing with" a toothbrush). Existing methods struggle to isolate action-specific visual evidence, leading to identical or mislocalized heatmaps for different actions on the same image.
Semantic Insufficiency: Current approaches rely on brief action phrases (e.g., "push") which lack the rich semantic detail required to distinguish semantically similar actions (e.g., "push" vs. "sit on" a motorcycle) that require grounding in distinct functional parts (handle vs. seat).

Although Large Vision-Language Models (LVLMs) encode rich action semantics and their textual outputs implicitly contain spatial cues, they are optimized for text generation rather than fine-grained spatial localization. Consequently, LVLMs do not directly output action-specific spatial maps, and their internal attention mechanisms often exhibit diffuse patterns attending to irrelevant background regions rather than target functional areas.

Methodology: TokAG
The authors propose TokAG, a zero-shot affordance grounding framework that leverages the latent semantic-spatial signals within pre-trained LVLMs without external supervision or task-specific training. The pipeline operates as follows:

Input and Generation: Given an egocentric image and a text prompt (e.g., "When people perform [Action] with [Object], which part of the [Object] is used?"), a frozen LVLM (e.g., Qwen3-VL) generates a sequence of output tokens.
Attention Aggregation: The framework extracts cross-modal attention maps between the generated output tokens and visual patches. Recognizing that affordance-relevant cues are distributed across different architectural layers and attention heads, the method aggregates attention weights across all layers and heads for each output token to capture distributed semantic-spatial cues.
Spatial-Aware Token Selection: To address the issue of diffuse attention and background noise, the method introduces a spatial constraint. Using a binary object mask (generated by CLIPSeg) for the target object, the system computes an "overlap score" for each aggregated attention map by measuring activation values strictly within the object region.
Token Selection and Heatmap Generation: The framework selects the single output token ( $t^*$ ) that yields the highest activation score over the target object area. The attention map corresponding to this selected token is upsampled to the original image dimensions to produce the final affordance heatmap.

Key Contributions

Zero-Shot Framework: TokAG is the first zero-shot affordance grounding framework that eliminates the need for affordance-specific supervision by exploiting token-level signals in LVLMs.
Spatial-Aware Token Selection: The authors propose a mechanism that systematically evaluates output tokens based on their object-focused activation, rather than relying on arbitrary attention maps or structural pruning. This transforms implicit semantic signals into precise spatial heatmaps.
Interpretability and Analysis: The paper provides an empirical analysis showing that the selected tokens often correspond to meaningful functional object parts (e.g., "blade," "seat," "handle"). Furthermore, the analysis indicates that stronger LVLM backbones (e.g., 32B vs. 2B) select more semantically relevant tokens, correlating with improved grounding performance.

Experimental Results
TokAG was evaluated on two benchmark datasets: AGD20K and HICO-IIF. The method was compared against state-of-the-art weakly supervised approaches and foundation-model-based baselines.

Performance on AGD20K: On the unseen split (Set 2), TokAG (using a 2B model) achieved an NSS of 1.514, a 10.7% improvement over the previous best weakly supervised method (Moon et al., 1.368). It also achieved the lowest KLD (1.060) and highest SIM (0.455).
Performance on HICO-IIF: TokAG achieved an NSS of 1.655 (with the 32B model), representing a 29.7% increase over existing weakly supervised baselines (1.234).
Comparison with Foundation Models: Direct prompting of LVLMs for coordinates or combining LVLMs with segmentation models (e.g., SAM 3, GroundingDINO) resulted in significantly lower performance. TokAG's approach of extracting spatial signals directly from attention maps proved superior to naive coordinate prediction or multi-stage pipelines.
Ablation Studies: Experiments confirmed that aggregating attention across all layers and heads outperforms selecting specific layers or heads. Additionally, the spatial-aware Top-1 token selection strategy outperformed heuristics like using the first/last token or averaging all tokens.

Significance and Claims
The paper claims that TokAG establishes a new state-of-the-art for zero-shot affordance grounding. Its primary significance lies in demonstrating that LVLMs, despite being text-generation models, contain sufficient implicit spatial knowledge to perform precise action localization when accessed via token-level attention analysis.

The authors note that while the framework is highly effective, it currently assumes affordances can be represented by a single semantic token. They acknowledge limitations in handling compositional affordances (requiring multiple regions, e.g., "pour from") and bimanual interactions. However, the work provides a foundational, training-free approach that significantly advances the field by bridging the gap between high-level semantic understanding and low-level spatial localization without requiring expensive pixel-level annotations.