💻 computer science

Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances

The paper introduces AFFORDMEM, a training-free framework that enhances 3D functional affordance grounding by leveraging cross-scene memory of annotated examples to guide vision-language models toward fine-grained regions and in-scene spatial memory to resolve complex relational queries, achieving state-of-the-art performance on the SceneFun3D benchmark without model fine-tuning.

Original authors: Qirui Wang, Jingyi He, Yining Pan, Xulei Yang, Shijie Li

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qirui Wang, Jingyi He, Yining Pan, Xulei Yang, Shijie Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to tidy up a messy room. You give it a simple command: "Close the second drawer from the top."

To a human, this is easy. We know what a drawer is, we know what a handle looks like, and we can count them to find the right one. But for a robot powered by current AI, this is a nightmare. The robot might get confused for two main reasons:

The "Zoom" Problem: The robot sees the whole drawer but doesn't know exactly which tiny part to grab. It might try to grab the entire wooden front of the drawer instead of the small metal handle.
The "Which One?" Problem: If there are three identical drawers, the robot gets lost. It doesn't know which one is "second from the top" because it can't see the whole room at once; it only sees one angle at a time.

The paper introduces a new system called AFFORDMEM to solve these problems. Instead of trying to teach the robot new skills from scratch (which takes a long time and lots of data), AFFORDMEM gives the robot a memory book and a mental map.

Here is how it works, using simple analogies:

1. The "Memory Book" (Cross-Scene Affordance Memory)

The Problem: When the robot sees a door handle, it doesn't know if it should grab the whole door or just the handle. It's like a child who has never opened a door before; they might try to push the entire wall.

The Solution: The robot has a Memory Book filled with pictures of handles, knobs, and buttons from thousands of other rooms it has "seen" before.

How it helps: When the robot sees a new handle, it flips through its Memory Book. It finds a picture of a similar handle and sees a bright red highlight showing exactly where a human would grab it.
The Analogy: Think of it like a teacher showing a student a photo of a "correct way to hold a pencil" before they try to write. The robot doesn't need to learn what a handle is from scratch; it just recalls, "Oh, I've seen this before! Humans grab right here."

2. The "Mental Map" (In-Scene Spatial Memory)

The Problem: The robot is looking at one drawer. It doesn't know there are two other drawers above it. It can't count "second from the top" if it can't see the whole stack.

The Solution: As the robot moves around the room, it builds a 3D Mental Map (like a game map or a blueprint). It doesn't just store pictures; it stores the location of every single handle it finds.

How it helps: When you say "second from the top," the robot looks at its Mental Map. It sees all three handles lined up vertically. It counts them on the map and says, "Ah, the one at this specific coordinate is the second one."
The Analogy: Imagine you are in a dark room with three identical light switches. You can't see them all at once. But if you have a blueprint of the room in your head, you know exactly where the second switch is relative to the first, even if you are currently looking at the wall with the first switch.

The Result

By combining these two tools:

The Memory Book tells the robot what to grab (the tiny handle, not the whole door).
The Mental Map tells the robot which one to grab (the second one, not the first).

The paper tested this on a dataset called SceneFun3D, which is a collection of real-world 3D room scans. The results showed that this "memory-based" approach worked significantly better than previous methods that didn't use these memory tricks.

Crucially, the paper emphasizes that this system is "training-free."

It doesn't need to be retrained on the new room.
It doesn't need a human to draw lines on the new room to teach it.
It just uses the "Memory Book" built from old data and the "Mental Map" built while looking at the new room to figure things out on the fly.

In short, AFFORDMEM makes robots smarter at following instructions by giving them a library of past experiences and a map of the current room, allowing them to find the exact right object to touch without needing a human to hold their hand.

Technical Summary: AFFORDMEM – Grounding by Remembering

Problem Statement

The paper addresses 3D functional affordance grounding, a task requiring an agent to not only recognize an object but to localize the specific, small 3D region that supports an interaction (e.g., the handle of a drawer, the button on a remote). The authors identify two primary challenges that existing training-free vision-language pipelines fail to address in combination:

Granularity Ambiguity: Actionable regions are often minute (e.g., a handle occupying fewer than 30×30 pixels in a high-resolution frame). Vision-Language Models (VLMs) prompted with text (e.g., "drawer handle") tend to localize the entire parent object (the drawer panel) rather than the specific operable subregion.
Instance Disambiguation: Scenes often contain multiple geometrically identical instances of the same object (e.g., three identical drawer handles). Resolving queries with spatial qualifiers (e.g., "the second handle from the top") requires global spatial reasoning across the entire scene, which single-view or frame-independent pipelines cannot provide.

Existing methods either treat frames as independent 2D bags, discarding 3D geometry, or focus on whole-object segmentation rather than fine-grained subregions.

Methodology: AFFORDMEM

The authors propose AFFORDMEM, a training-free framework that grounds 3D functional affordances by maintaining memory at two distinct levels. The system processes a posed RGB-D sequence and a natural language query without fine-tuning the underlying models or annotating the target scene.

1. Functional Query Decomposition

The input query is parsed into a structured triple:

$\ell_{ctx}$ : The reference object label (e.g., "drawer").
$\ell_{int}$ : The interactive element label (e.g., "handle").
$\sigma$ : The spatial relational descriptor (e.g., "second from the top").

2. Cross-Scene Affordance Memory (Addressing Granularity)

This component addresses the granularity ambiguity by providing a category-level geometric prior.

Memory Construction: A memory bank is built from previously annotated source scenes. For each semantic category, the system projects annotated 3D affordance points into 2D image planes using the source camera poses. It filters these projections for depth consistency to suppress false positives at occlusion boundaries.
Recall & Prompting: At query time, the system retrieves the top- $K$ most informative masked images (showing where affordances typically appear on the object) for the target category. These images are provided as visual prompts to a frozen VLM alongside the current frame.
Adversarial Prompting: The VLM is instructed to output paired bounding boxes: a "positive" box for the directly operable core and a "negative" box for visually similar but non-operable regions (e.g., the mounting base). This steers the model away from coarse object-level segmentation.
3D Fusion: The resulting fine-grained 2D masks are back-projected and fused into a 3D candidate pool using multi-view voting and clustering (DBSCAN).

3. In-Scene Spatial Memory (Addressing Disambiguation)

This component addresses instance disambiguation by organizing the 3D candidate pool into a structured scene graph.

Scene Graph Construction: Each 3D candidate instance is a node annotated with its 3D centroid, Axis-Aligned Bounding Box (AABB), and semantic type. Nodes are linked to their parent objects to encode part-object relationships.
Global Reasoning: The graph is serialized and presented to a Large Language Model (LLM) alongside a schematic, top-down orthographic render of the scene (where all candidates are visible without occlusion) and per-node RGB crops.
Selection: The LLM reasons over the global layout to resolve spatial qualifiers (e.g., "second from the top") and selects the correct node ID, which corresponds to the final 3D mask prediction.

Key Contributions

Cross-Scene Affordance Memory: A category-level visual memory bank that recalls geometric priors to guide frozen VLMs toward small operable subregions, overcoming the tendency to select enclosing objects.
In-Scene Spatial Memory: A structured spatial memory that accumulates 3D candidates and their relations, enabling reliable selection under ordinal, relational, and long-distance spatial qualifiers via LLM reasoning.
Training-Free State-of-the-Art: The framework requires no model fine-tuning and no target-scene annotation, achieving new state-of-the-art performance among training-free methods on the SceneFun3D benchmark.

Experimental Results

The method was evaluated on SceneFun3D, a large-scale benchmark for 3D functional affordance segmentation.

Performance: AFFORDMEM outperformed the prior training-free state-of-the-art (Fun3DU) by +3.23 AP50 on Split 0 and +3.7 AP50 on Split 1.
Comparison to Training-Based Methods: On Split 0, AFFORDMEM surpassed the training-based method AffordBot by a significant margin in AP25 (41.66 vs. 24.76), indicating superior localization of operable subregions despite lacking scene-specific training.
Ablation Studies:
- Removing the In-Scene Spatial Memory (scene graph) caused the largest performance drop (7.18 AP50), confirming its necessity for resolving spatial qualifiers.
- Removing Cross-Scene Affordance Memory (adversarial prompting) reduced AP25 by 2.92 points, validating its role in fine-grained localization.
- A diagnostic "Fun3DU-style" pipeline using the same backbone models (Qwen3-VL and SAM3) but without the memory modules achieved significantly lower performance (7.14 AP50), demonstrating that the gains stem from the proposed memory mechanisms rather than backbone improvements.

Significance and Claims

The paper claims that 3D geometry encodes the specific signals missing in current pipelines: category-level spatial priors for granularity and global instance layouts for disambiguation. By treating these as distinct "memory failures" and addressing them with dedicated geometric memory mechanisms, AFFORDMEM bridges the gap between object recognition and precise functional interaction.

The authors emphasize that their approach is modest regarding generalization: the current system relies on exact string matching between interaction labels and the memory index, meaning it cannot generalize to unseen categories without prior annotation. They identify future work in embedding-based retrieval for unseen categories and extending the framework to dynamic scenes. The primary contribution is a robust, training-free framework that sets a new baseline for 3D functional affordance grounding by effectively leveraging memory to solve granularity and spatial ambiguity.