RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

1. Problem Statement

Robots operating in unstructured environments require the ability to understand object affordances—specifically, where to interact (contact points) and how to interact (action directions). Existing approaches face two main limitations:

• Retrieval-based methods: Rely on finding a single "top-1" match from a memory bank. They are fragile due to data sparsity; if the single best match is slightly misaligned, the prediction fails. They also struggle with "coverage gaps" where no semantically similar instance exists in memory.
• Training-based methods (Large-scale models): While they learn general patterns, they often fail to localize precise contact points or predict reliable post-contact action directions for unseen object categories. They frequently require massive datasets and still struggle with zero-shot generalization in real-world scenarios.

The core challenge is achieving robust generalization for both static contact localization and dynamic action direction prediction with limited training data (data scarcity).

2. Methodology: RAAP Framework

The authors propose Retrieval-Augmented Affordance Prediction (RAAP), a unified framework that decouples affordance prediction into two components with distinct uncertainty characteristics:

1. Static Affordance (Contact Point): Governed by geometric correspondence.
2. Dynamic Affordance (Action Direction): Inherently ambiguous and requiring evidence aggregation.

A. Affordance Memory Construction

RAAP builds a visual memory $R$ containing tuples of $(I_g, f, T, A_{2D})$, where $I_g$ is a segmented object image, $f$ is a CLIP appearance embedding, $T$ is the task label, and $A_{2D}$ is the 2D affordance annotation (contact point $c_{2D}$ and action direction $a_{2D}$). The memory is populated from subsets of the DROID and HOI4D datasets.

B. Static Affordance Transfer (Dense Correspondence)

For the contact point, RAAP uses a top-1 retrieval strategy:

• It retrieves the single most similar reference image based on CLIP similarity.
• It employs Stable Diffusion (SD) features to establish dense pixel-to-pixel correspondence between the query image and the top-1 reference.
• The contact point is transferred by finding the location in the query image with maximal cosine similarity to the annotated contact point in the reference. This leverages the high-resolution geometric precision of SD features.

C. Dynamic Affordance Prediction (Retrieval-Augmented Alignment)

For action direction, RAAP aggregates cues from Top-K references ($K > 1$) to reduce ambiguity. This is achieved via a novel Retrieval-Augmented Alignment Model:

• Action-Conditioned Encoding: Reference images are encoded using a shared SigLIP-2 backbone. The associated action vectors are projected via an MLP and used to modulate the image features via FiLM (Feature-wise Linear Modulation). This allows the model to understand the visual context in the context of the specific action.
• Dual-Weighted Attention: To fuse information from multiple references, the model uses a cross-attention mechanism with a dual-weighting strategy:
  1. Learned Semantic Relevance: A gating network computes weights based on the semantic similarity between the query and reference global features.
  2. Pre-computed Visual Similarity: Weights based on the initial CLIP retrieval scores.
  • The final attention weights combine both signals, allowing the model to emphasize task-relevant priors while suppressing noisy or misaligned examples.
• Prediction: The fused tokens are processed by a Transformer encoder, and the final action direction is regressed from a [CLS] token.

D. Execution (3D Lifting)

The predicted 2D affordance is lifted to 3D using camera intrinsics and object point clouds. The robot selects a grasp pose near the predicted contact point and executes the action by moving the end-effector along the predicted 3D displacement vector.

3. Key Contributions

1. Unified Paradigm: RAAP successfully unifies retrieval and training-based approaches, addressing the sparsity of retrieval and the generalization limits of pure training.
2. Decoupled Design: It explicitly separates static contact localization (solved via dense correspondence) from dynamic action prediction (solved via multi-reference alignment), recognizing their different uncertainty profiles.
3. Novel Alignment Model: Introduces a retrieval-augmented alignment module with FiLM modulation and dual-weighted attention, enabling robust aggregation of multiple references even with limited data.
4. Data Efficiency: The framework achieves strong performance with as few as tens of samples per task, enabling zero-shot manipulation in both simulation and the real world.

4. Experimental Results

The method was evaluated on DROID, HOI4D, and real-world robotic platforms (Franka Emika).

• Quantitative Performance (Task-Aware Prediction):

  • On the DROID dataset, RAAP ($K=3$) achieved a Mean Angular Error (MAE) of 32.55°.
  • This represents a ~50% reduction in error compared to the retrieval-only baseline (RAM, 62.84°) and the large-scale diffusion baseline (A0, 74.81°).
  • RAAP significantly outperformed baselines on "Open/Close" tasks, where single-reference methods often fail to predict correct directions.

• Real-World Generalization:

  • Unseen Objects: RAAP achieved success rates of 85% (Open Drawer) and 90% (Pickup Bowl), significantly outperforming RAM and A0 (which often failed due to poor contact localization).
  • Cross-Category: RAAP achieved 100% success on "Close Cabinet" (trained on drawers/microwaves) and 85% on "Pickup Plate."
  • Simulation: In MuJoCo, RAAP achieved 85% success on transferring "Pickup Mug" skills to "Pickup Kettle," outperforming RAM (70%) and A0 (10%).

• Ablation Studies:

  • Removing the dual-weighting mechanism (using only gating or only similarity) led to significant performance drops, confirming the necessity of combining semantic and visual cues.
  • Performance peaked at $K=3$; too few references ($K=0, 1$) left the model vulnerable to retrieval errors, while too many ($K=4$) introduced noise.

5. Significance

RAAP represents a significant step forward in data-efficient robotic manipulation. By acknowledging that different aspects of affordance (static vs. dynamic) require different handling strategies, it overcomes the "sparsity vs. coverage" trade-off inherent in previous methods.

• Practical Impact: It enables robots to perform fine-grained tasks on unseen objects and categories without requiring extensive retraining or large-scale real-world data collection.
• Scalability: The ability to function with minimal training samples makes it highly applicable to real-world deployment where data collection is costly.
• Robustness: The dual-weighted attention mechanism provides a robust way to handle visual and geometric variations, making it suitable for unstructured environments.

The paper concludes that while RAAP currently focuses on short-horizon tasks, its success in zero-shot generalization lays the groundwork for more complex, long-horizon robotic interactions.