Technical Summary: Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

1. Problem Statement

Affordance grounding, the task of localizing specific interaction regions on an object given a natural language description (e.g., "where to hold a cup"), is a fundamental capability for embodied agents. However, progress in this field is severely bottlenecked by data limitations:

• Scalability and Cost: Manual annotation of affordance regions is prohibitively expensive and time-consuming.
• Limited Scope: Existing datasets are constrained to a narrow set of predefined object and affordance categories (closed-vocabulary), failing to capture the open-ended nature of real-world interactions.
• Ambiguity: Affordance regions often have indistinct boundaries and vary significantly across object instances, making precise localization difficult without diverse supervision.

Current 3D and 2D datasets rely heavily on manual labeling or weak supervision from videos, which introduces noise and bias due to occlusions and limited viewpoints. There is a critical lack of large-scale, open-vocabulary datasets that can support generalization to unseen object-affordance combinations.

2. Methodology

2.1 The Affogato Framework and Automated Pipeline

The authors introduce Affogato, a scalable framework centered on Affogato-750K, a large-scale dataset generated entirely without human labeling. The core innovation is a fully automated annotation pipeline that orchestrates multiple foundation models to generate 750K affordance annotations across 150K 3D objects.

The pipeline operates in three stages (Figure 2):

1. Open-Vocabulary Query Generation: Using Gemma3 (a vision-language model), the system generates five natural language affordance queries per object based on multi-view renderings. To ensure functional relevance, the pipeline employs Chain-of-Thought (CoT) prompting, forcing the model to first identify the object's semantic class and then generate queries conditioned on its functionality, rather than relying solely on shape.
2. Interaction Point Prediction: The generated queries are fed into Molmo (a multimodal model) to predict precise pixel coordinates representing the most likely interaction point in each 2D view.
3. Heatmap Generation and Aggregation: The predicted points serve as prompts for MobileSAM to generate 2D segmentation masks. The system adopts a "smallest-mask" selection strategy to isolate part-level structures (e.g., handles, buttons) rather than whole-object silhouettes. These 2D masks are projected onto the 3D object surface using depth and camera parameters. A multi-view voting mechanism aggregates predictions from 25 viewpoints, reinforcing consistent regions and suppressing errors from individual views to produce robust 3D affordance heatmaps.

2.2 The Espresso Models

To isolate the contribution of the dataset from architectural biases, the authors propose Espresso-3D and Espresso-2D. These are minimal, symmetric models designed for 3D point clouds and 2D images, respectively.

• Architecture: Both models consist of a modality-specific visual encoder (PartField for 3D, DINOv2 for 2D), a text encoder (Recap-CLIP for 3D, CLIP for 2D), and a text-conditioned heatmap decoder.
• Design: Unlike prior methods that use learnable prompt embeddings, Espresso uses text embeddings directly as queries in the decoder. This design inherently supports open-vocabulary grounding without predefined categories.

3. Key Contributions

1. Affogato-750K Dataset: A large-scale dataset comprising 750K affordance annotations over 150K 3D objects. It significantly surpasses existing datasets in scale and diversity, covering over 450 object classes and 350 affordance types (compared to ~23 classes in prior 3D datasets). It includes a 5K human-verified test split for reliable evaluation.
2. Automated Data Generation Pipeline: A novel pipeline that leverages foundation models (Gemma3, Molmo, MobileSAM) to generate high-quality, open-vocabulary affordance data at scale, eliminating the need for manual annotation.
3. Unified Baseline Models (Espresso): Simple yet effective models (Espresso-3D and Espresso-2D) that demonstrate the efficacy of the dataset. The authors show that pretraining on Affogato-750K yields substantial gains for both their models and prior state-of-the-art methods.

4. Experimental Results

4.1 3D Affordance Grounding

• LASO Benchmark: Pretraining on Affogato-750K consistently improves performance on the LASO test split, particularly in the unseen setting (zero-shot generalization). For instance, PointRefer gains +4.0% aIoU on unseen classes, and Espresso-3D gains +2.1% aIoU.
• Affogato-750K Test Split: On the authors' own diverse test split, Espresso-3D outperforms OpenAD and PointRefer across in-domain and cross-domain settings. The results highlight the difficulty of generalizing to the broad distribution of Affogato-750K, with significant performance gaps remaining even for top methods.
• Cross-Domain Transfer: Training on "Daily-Used" objects transfers well to "Furnitures," but the reverse is less effective, attributed to the larger volume and diversity of the Daily-Used training set.

4.2 2D Affordance Grounding

• AGD20K Benchmark: Espresso-2D pretrained on Affogato-750K achieves state-of-the-art results in both zero-shot and fully supervised settings on AGD20K.
• Generalization: In zero-shot evaluation, Espresso-2D achieves 40.2 SIM on seen splits and 37.6 SIM on unseen splits, significantly outperforming baselines like LISA-7B and Molmo+SAM2.
• Supervised Gains: Even when fine-tuned on AGD20K, models pretrained on Affogato-750K outperform those trained from scratch, confirming the transferability of the learned affordance concepts.

4.3 Analysis

• Scale Effect: Performance improves monotonically with the size of the pretraining subset (50K to 750K), confirming that the gains are driven by data scale and diversity.
• Resolution: Training on high-resolution point clouds (16,384 points) yields better results than low-resolution (2,048 points), emphasizing the importance of fine-grained geometric details.
• Synthetic-to-Real: Despite being trained on clean Objaverse renderings, Espresso-2D successfully localizes affordances in cluttered, real-world robot workspace images (Open X-Embodiment), demonstrating strong domain generalization.

5. Significance and Claims

The paper claims that Affogato provides a scalable solution to the data bottleneck in affordance grounding. By automating the generation of open-vocabulary annotations, the framework enables the creation of datasets that are orders of magnitude larger and more diverse than previously possible.

The authors emphasize that Affogato-750K serves as a broadly transferable supervision source. The consistent improvements across different architectures (Espresso, PointRefer, OpenAD) and modalities (2D and 3D) suggest that the dataset captures fundamental, generalizable representations of object functionality. While the framework currently focuses on localizing contact points, the authors note that predicting finer-grained dynamics (e.g., force, temporal sequences) remains an open challenge for future work. The work establishes a new benchmark for embodied AI, moving the field from closed-vocabulary constraints toward true open-vocabulary understanding.