💻 computer science

EgoAfford: Task-Oriented Affordance Grounding via Egocentric Referring Segmentation

The paper introduces EgoAfford, a benchmark and dataset for task-oriented affordance grounding in egocentric views, along with EgoLens, a specialized multimodal model that jointly performs next-step planning and role-specific segmentation for multi-step tabletop tasks.

Original authors: Xinyuan Guan, Feifan Chen, Xinyu Zhan, Fu-Cheng Zhang, Cewu Lu, Lixin Yang

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyuan Guan, Feifan Chen, Xinyu Zhan, Fu-Cheng Zhang, Cewu Lu, Lixin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a sandwich. You don't just want the robot to know what a "knife" looks like; you need it to understand how to use that knife to spread butter, what to spread it on, and where to put the finished slice. This is the world of affordance, a fancy word for "what an object allows you to do." Think of a doorknob: its shape affords turning. A cup affords holding liquid. For a long time, robots have been pretty good at spotting these simple, one-off actions, like "grab the handle." But real life is messy and multi-step. Making a sandwich isn't just one grab; it's a whole story involving planning, picking up the right tool, and knowing exactly where to put things next. The big question scientists are asking is: Can we teach robots to not just see objects, but to understand the story of a task, figure out the next move, and point out the tiny, specific parts of an object needed for that move?

Enter EgoAfford, a new project that tries to solve this puzzle by giving robots a "first-person" view of the world, just like we see it. The researchers built a massive digital playground with about 15,500 images of tabletop scenes, covering 2,000 different multi-step tasks. They also created a "real-world" test set with 102 photos taken by humans to see if the robot can handle the messy reality of a real kitchen. The goal? To see if a computer can look at a picture, read a goal like "make tea," and then do two things at once: 1) Write down the rest of the recipe (the plan), and 2) Draw a mask over the exact part of the teapot you need to pour from, the spoon you need to stir with, and the cup you need to pour into.

To tackle this, the team introduced EgoLens, a smart AI model designed specifically for this job. Think of EgoLens as a robot chef's apprentice who is learning to "read" a scene. Unlike other models that might guess the whole object or get confused by complex instructions, EgoLens is trained to break a task down into its smallest functional pieces. It doesn't just say "that's a pot"; it says, "I need to grab the spout of that pot to pour, and I need to hold the handle of the spoon to stir." The paper shows that EgoLens is currently the best at this specific type of thinking, outperforming other big AI models and even commercial tools that try to combine planning and vision.

However, the researchers are careful to note that this isn't a magic solution yet. While EgoLens is great at the "digital" images they generated, there is still a gap when it comes to real-world photos, suggesting that the robot still needs more practice with the unpredictable nature of real life. The study suggests that by combining the ability to plan the next steps with the ability to pinpoint the exact functional part of an object, we can get robots much closer to being helpful helpers in our daily lives. It's a significant step forward in teaching machines to understand not just what things are, but how to use them to get a job done.

Technical Summary: EgoAfford – Task-Oriented Affordance Grounding via Egocentric Referring Segmentation

Problem Formulation

Current affordance grounding research has largely focused on elemental actions, localizing functional parts (e.g., a blade for cutting) for single objects or single-step instructions. However, extending this to complex, multi-step tabletop tasks reveals two critical limitations in prevailing formulations:

Single-Target Focus: Existing methods typically ground one action-conditioned region without explicitly assigning semantic roles (direct object, instrument, destination) to multiple participating objects.
Single-Step Scope: Even recent bimanual or scene-level formulations predict affordances for a single given action rather than inferring a remaining procedure conditioned on task progress.

EgoAfford addresses these gaps by formulating task-oriented affordance grounding as a joint problem: given an egocentric tabletop image and a high-level natural language task, a model must (i) generate the remaining plan of action steps and (ii) segment the functional parts of the next action step into three specific roles:

Direct Object ( $M_o$ ): The object being manipulated.
Instrument ( $M_s$ ): The tool used to act on the object (absent for bare-hand actions).
Destination ( $M_d$ ): The target location or container for transfer (absent for in-place actions).

Crucially, these masks must be part-level (e.g., the spout of a pot, not the whole pot) and handle absent components (annotated as all-zero masks).

Methodology

The EgoAfford Benchmark

The authors introduce a comprehensive benchmark consisting of a dataset and a reference model.

Dataset Construction: The dataset contains approximately 15,537 human-verified egocentric images derived from 2,000 generated multi-step scenes. A generative pipeline using commercial VLMs (e.g., GPT-5.5, Gemini 3) creates task seeds, decomposes them into steps, and synthesizes semantically aligned images. These are followed by a rigorous human verification and mask correction process.
EgoAfford-Real: To evaluate generalization, a manually captured test set of 102 images spanning 26 tasks is included, disjoint from the generated scenes.
Annotation Protocol: Annotations define three role-specific masks per step. The dataset supports multi-branch tasks where multiple valid next actions exist, requiring models to select the most appropriate action and ground its components.

The EgoLens Reference Model

To establish a strong baseline, the authors present EgoLens, a 3B-parameter Multi-Modal Large Language Model (MLLM) designed for this joint task.

Architecture: Built upon the LENS architecture with a Qwen2.5-VL 3B backbone. It utilizes learnable grounding queries passed through three parallel, role-specific mask decoders (one for each component: $M_o, M_s, M_d$ ). This architectural design ensures semantic binding between masks and their roles without relying solely on text generation.
Training Strategy: The model is trained with a unified prompt template. It generates a Chain-of-Thought (CoT) planning trace (teacher-forced during training) followed by a structured answer containing the remaining plan and the three component masks. The loss function combines language modeling loss (supervising only the structured answer tokens) and segmentation loss (BCE + Dice for non-empty masks, BCE only for empty masks).
Inference: The model autoregressively generates the full plan and the three component masks in a single forward pass based on the image and task description.

Key Contributions

Task-Oriented Formulation: The paper defines multi-step task-oriented affordance grounding as a joint remaining-plan generation and part-level grounding task, instantiating it in the EgoAfford benchmark.
EgoLens Model: A reference MLLM that simultaneously generates the remaining plan and predicts three role-specific component masks, addressing the challenge of coupling planning with fine-grained grounding.
Comprehensive Evaluation: Extensive evaluations on both generated and real-world observations, characterizing the specific challenges of next-step inference and action-role-conditioned part grounding.

Experimental Results

The paper evaluates recent referring-segmentation MLLMs (Sa2VA, UniPixel, LENS, OMG-LLaVA), commercial VLM–SAM2 pipelines, and EgoLens.

Full Task Performance: EgoLens establishes strong reference performance, achieving 0.700 gIoU and 0.486 cIoU on the generated test set. This significantly outperforms the strongest commercial VLM pipeline (GPT-5.5 + SAM2, 0.476 gIoU) and open-source baselines, which often fail to return all three role-specific masks in a single pass.
Real-World Generalization: On EgoAfford-Real (zero-shot transfer), EgoLens achieves 0.666 gIoU and 0.455 cIoU, outperforming the GPT-5.5 + SAM2 pipeline by 9.7 points in gIoU. This suggests EgoLens provides more consistent sample-level predictions.
Disentangling Inference and Grounding: When provided with a reference next step (removing the planning burden), EgoLens-Seg achieves 0.839 gIoU. The drop in performance when the reference step is removed highlights that next-step inference remains a substantial source of error.
Component Analysis: EgoLens demonstrates robust localization for non-empty targets (NE-mIoU of 0.668 for Direct Objects, 0.614 for Instruments, and 0.568 for Destinations), confirming performance is not solely driven by recognizing absent components.

Significance and Claims

The paper posits that EgoAfford and EgoLens provide a foundation for jointly studying perception and planning in multi-step tabletop tasks. The authors claim that:

Complementary Challenges: The benchmark highlights the complementary difficulties of next-step inference (planning) and action-role-conditioned part grounding.
Role-Specific Grounding: Moving beyond single-object formulations to explicitly model direct objects, instruments, and destinations captures how multiple objects participate in a task.
Future Direction: The work suggests a path toward connecting role-structured grounding with embodied execution, where identified components serve as explicit arguments for reusable manipulation skills (e.g., pour(source, destination)), offering an agentic alternative to end-to-end World Action Models.

The authors acknowledge limitations, noting that the dataset is largely generated (lacking the visual fidelity and clutter of natural environments) and that static-image evaluation does not capture motion feasibility or closed-loop success. They also note that EgoLens is trained on a single valid path, potentially limiting its ability to handle alternative solutions.