Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Models

This paper demonstrates that affordance reasoning in Vision Foundation Models can be achieved in a zero-shot, training-free manner by fusing DINO's inherent geometric part structures with Flux's verb-conditioned interaction priors, thereby establishing geometric and interaction perception as the fundamental, composable building blocks of affordance understanding.

The Gist

Imagine you are looking at a chair. A basic camera just sees a pile of pixels: four legs, a seat, a backrest. But a smart visual system doesn't just see "chair"; it sees "sittable," "climbable," and "graspable." This ability to see how an object can be used is called Affordance.

For a long time, scientists tried to teach computers this by showing them millions of labeled pictures (e.g., "This red pixel is where you hold the knife"). But this is like teaching a child to swim by only showing them diagrams of water; it's slow, expensive, and the child might not know how to swim in a different pool.

This paper asks a simpler, more profound question: "Do computers already know how to swim? We just haven't asked them the right way."

The authors argue that to truly understand how to use an object, a computer needs two specific "superpowers":

1. The Architect's Eye (Geometry): Seeing the physical shape and parts of an object.
2. The Actor's Eye (Interaction): Imagining how a human body would move to touch or use those parts.

Here is how they proved this and combined these powers, explained through a few analogies.

1. The Two Superpowers

The Architect: "The Shape Detective"

The researchers looked at a type of AI called DINO. Think of DINO as a master architect who looks at a messy pile of bricks and instantly sees the hidden structure: "That's a handle," "That's a blade," "That's a rim."

• The Discovery: They found that DINO naturally breaks objects down into these functional parts without ever being taught to do so. It sees the "handle" of a mug not because it knows the word "mug," but because it recognizes the shape of a handle.
• The Metaphor: If you handed DINO a picture of a weird alien tool, it would still point out, "You can grip this part," because it understands the geometry of gripping.

The Actor: "The Movie Director"

The researchers then looked at a different type of AI called Flux (a generative model that creates images from text). Think of Flux as a movie director who knows exactly where to place the actors.

• The Discovery: When the researchers asked Flux to "Imagine a person holding a knife," the AI's internal "attention map" (a mental spotlight) automatically lit up the handle of the knife. When they said "Imagine a person drinking from a cup," the spotlight moved to the rim.
• The Metaphor: Flux didn't need to be told where the handle is. It just knew that if a human is going to "hold" something, their hand must go there. It has an innate sense of interaction.

2. The Magic Trick: Mixing the Architect and the Actor

The big breakthrough of this paper is realizing that these two AIs are like a Lock and Key.

• DINO (The Architect) knows where the parts are (the lock).
• Flux (The Actor) knows where the action happens (the key).

Usually, to get a computer to find affordances, you have to train it for years with expensive data. But the authors asked: "What if we just let them talk to each other?"

They created a simple, training-free recipe:

1. Take a picture of an object.
2. Ask DINO to highlight the physical parts (e.g., "Here is the handle").
3. Ask Flux to imagine an action (e.g., "Hold this") and see where its mental spotlight lands.
4. Fuse them: Where DINO says "This is a handle" AND Flux says "This is where a hand goes," you get the perfect answer.

3. The Result: Zero-Shot Magic

The result was surprising. By simply combining these two pre-existing "brains," they created a system that could guess affordances for objects it had never seen before, without any new training.

• The Analogy: Imagine you want to know how to use a strange, futuristic gadget.
  • Old Way: You spend 10 years studying manuals and watching people use it.
  • This Paper's Way: You show the gadget to a Geometer (who sees the shape) and a Choreographer (who knows how humans move). You ask them to work together. Instantly, they point to the button you should press.

Why This Matters

This changes the game for robotics and AI. Instead of building a new, expensive robot brain for every new task, we can just assemble the right tools we already have.

• Geometry gives us the "what" (the parts).
• Interaction gives us the "how" (the action).

When you put them together, the computer doesn't just "see" the world; it understands how to play in it. It's a shift from "teaching by rote" to "combining innate talents," making AI more adaptable, cheaper to build, and closer to how humans naturally understand the world.

Technical Summary

1. Problem Statement

Visual affordance refers to the possibilities for action that objects in a scene offer to an agent (e.g., a chair affords sitting, a handle affords grasping). While recent advances in Vision Foundation Models (VFMs) have improved general visual understanding, it remains unclear what internal capabilities enable a model to truly "understand" affordance.

Existing approaches to affordance learning generally fall into three paradigms, each with limitations:

• Fully Supervised: Learns geometric patterns from dense pixel-level annotations but lacks scalability and generalization to new categories.
• Weakly Supervised: Infers affordance from indirect cues (e.g., human-object interaction videos) but often suffers from coarse spatial localization.
• Open-Vocabulary: Leverages semantic associations from image-text pretraining but often conflates function with category-level semantics, lacking grounded interaction cues.

The core research question is: What are the fundamental capabilities required for affordance understanding, and how are they manifested within pre-trained VFMs without task-specific training?

2. Methodology

The authors propose a dual-dimensional framework decomposing affordance into two complementary capacities:

1. Geometric Perception: Identifying the structural parts of objects that enable interaction (e.g., handles, blades, seats).
2. Interaction Perception: Modeling how an agent's actions engage with those specific parts (e.g., "grasping" a handle).

To validate this hypothesis, the authors conducted a systematic probing of diverse VFMs (including DINOv2, DINOv3, CLIP, SAM, Stable Diffusion, and Flux) and developed a training-free fusion pipeline.

A. Probing Geometric Perception (Discriminative Models)

• Linear Probing: The authors froze the backbones of various VFMs and trained a simple linear head to predict affordance maps on the UMD dataset.
• Key Finding: Models with stronger inherent geometric awareness (specifically DINOv3) achieved higher affordance segmentation accuracy.
• Representation Analysis: Using PCA, they discovered that DINO models organize visual features into part-level structures (e.g., isolating handles or rims) rather than just semantic categories. This part-level decomposition is crucial for actionable perception.
• Semantic vs. Geometric: They demonstrated that while DINO captures geometry, it is sometimes entangled with semantics. To isolate pure geometric prototypes, they used PCA to extract interpretable part-level bases.

B. Probing Interaction Perception (Generative Models)

• Hypothesis: Generative models, trained to synthesize interactions, might inherently encode "verb-conditioned" spatial priors.
• Mechanism: The authors analyzed the cross-attention maps of Flux (a generative diffusion model). They found that verb tokens (e.g., "hold," "cut," "drink") consistently attend to the specific contact regions on objects relevant to that action.
• Extraction: Using Flux Kontext (an image editing model), they extracted these verb-conditioned attention maps as zero-shot interaction guides. These maps localize interaction regions without any explicit affordance supervision.

C. The Fusion Framework (Geometry + Interaction)

The authors proposed a training-free, zero-shot fusion pipeline to combine these two dimensions:

1. Geometry Source: Use DINOv3 to generate dense feature maps. Apply PCA to extract part-level geometric prototypes.
2. Interaction Source: Use Flux Kontext to generate verb-conditioned cross-attention maps indicating probable action regions.
3. Fusion:
   • Use the object attention from Flux to crop a Region of Interest (ROI) from the DINOv3 feature map.
   • Compute Normalized Scanpath Saliency (NSS) between the geometric PCA components and the verb attention map.
   • Select the geometric component that best aligns with the verb attention and fuse them to produce the final affordance mask.

3. Key Contributions

1. Dual-Dimensional Framework: A unified theoretical view that affordance understanding emerges from the coupling of geometric structure (object parts) and interaction dynamics (agent actions).
2. Systematic Probing of VFMs: The first comprehensive analysis revealing that:
   • Discriminative models (like DINO) inherently encode part-level geometric structures.
   • Generative models (like Flux) inherently encode verb-conditioned spatial interaction priors via cross-attention.
3. Training-Free Affordance Estimation: A novel method that mines internal cross-attention maps from generative models as spatial priors, achieving competitive results without fine-tuning.
4. Zero-Shot Performance: Demonstrated that fusing these two pre-trained capabilities yields affordance estimation competitive with weakly-supervised methods, proving that these capabilities are composable elements within existing models.

4. Results

• Geometric Correlation: A strong positive correlation was found between a model's geometric awareness (measured by Probing3D) and its affordance segmentation performance (mIoU on UMD). DINOv3 outperformed semantic-heavy models like CLIP and SAM in this regard.
• Interaction Priors: Verb-conditioned attention maps from Flux successfully localized contact regions (e.g., hands on a keyboard, blades on a knife) with high spatial consistency, even in cases where the image generation failed.
• Fusion Performance (AGD20K Dataset):
  • The proposed Interaction × Geometry method achieved an NSS of 1.090 and SIM of 0.326.
  • This performance is competitive with state-of-the-art weakly-supervised methods (e.g., LOCATE: NSS 1.157) and significantly outperforms using interaction cues alone (NSS 1.050).
  • The fusion reduced KLD (Kullback-Leibler Divergence), indicating sharper and more confident predictions by suppressing implausible regions.

5. Significance and Future Directions

• Mechanistic Insight: The paper provides a mechanistic account of how perception grounds action, showing that affordance is not a single learned task but a composition of geometric and interactional primitives.
• Paradigm Shift: It moves the field from training task-specific models to assembling innate capabilities from pre-trained foundation models. This offers a scalable, data-efficient path for embodied AI.
• Generative Models as Priors: It highlights the untapped potential of generative models (specifically their attention mechanisms) as rich sources of spatial priors for reasoning tasks.
• Future Work: The authors suggest that video generative models could further unify these dimensions by modeling 3D scene geometry and interaction dynamics simultaneously, offering a complete foundation for dynamic affordance reasoning.

In summary, this work demonstrates that by simply fusing the geometric part-structure of discriminative models with the verb-conditioned attention of generative models, one can achieve robust, zero-shot affordance reasoning, fundamentally changing how we approach visual understanding in embodied intelligence.