💻 computer science

ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Models

ORACLE-Grasp is a zero-shot framework that leverages Large Multimodal Models as semantic oracles to guide affordance-aligned robotic grasping in unstructured environments, eliminating the need for task-specific training while achieving high success rates through a structured, iterative decision process and depth-based refinement.

Original authors: Avihai Giuili, Rotem Atari, Avishai Sintov

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Avihai Giuili, Rotem Atari, Avishai Sintov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot walking into a messy room for the first time. You see a strange object on the table. You don't know what it is, you've never seen it before, and no one has told you how to pick it up.

In the past, robots would freeze. They needed a manual, a 3D blueprint, or thousands of photos of that specific object to figure out where to grab it. If they tried to grab a hammer by the head instead of the handle, they might break it or fail to use it.

ORACLE-Grasp is a new "brain" for robots that solves this problem. It uses a super-smart AI (called a Large Multimodal Model) to act like a wise, experienced human who can look at any object and instantly know the best way to hold it, even if they've never seen it before.

Here is how it works, broken down into simple steps with some fun analogies:

1. The "Oracle" (The Wise Guide)

Think of the AI model as an Oracle (like a fortune teller who knows everything). But instead of predicting the future, this Oracle predicts the best way to hold things.

The Problem: Oracles are great at talking and understanding concepts, but they are terrible at pointing with their finger. If you ask an Oracle, "Where is the handle?" it might say, "It's somewhere on the right side," but that's too vague for a robot arm.
The Solution: ORACLE-Grasp doesn't let the Oracle guess randomly. Instead, it plays a game of "Hot or Cold" on a grid.

2. The Game of "Hot or Cold" (The Dual-Prompt Strategy)

Instead of asking the Oracle to draw a box around the handle, the system cuts the picture of the object into a grid of tiles (like a checkerboard).

Step 1: The Context (The "What"): First, the system asks the Oracle: "What is this object, and what part of it is meant to be held?"
- Analogy: Imagine you show a picture of a weird tool to a friend and ask, "What is this?" They say, "It looks like a weird drill." Then you ask, "How would you hold it?" They say, "By the handle, obviously." This gives the robot the context.
Step 2: The Grid Search (The "Where"): Now, the system shows the Oracle the grid and asks: "Based on what you just said, which square on this grid contains the handle?"
- Analogy: Instead of asking the friend to point to the exact millimeter, you ask, "Is it in square A4 or B2?" The friend picks the right square.

3. The "Chain of Thought" (Thinking Before Acting)

The system forces the Oracle to explain why it picked that square before it actually picks it.

Analogy: It's like a teacher asking a student, "Don't just give me the answer; show your work." The Oracle has to say, "I picked square A4 because it looks like a handle, and handles are usually where you hold things."
Why this matters: This stops the robot from grabbing the wrong part (like the sharp blade of a knife) because the AI has to logically justify its choice first.

4. The "Zoom Lens" (Refining the Grip)

Sometimes, the grid is too coarse, or the object is tricky (like a hollow kettle handle).

The Fix: The system takes the best guess, zooms in on that specific area, and uses a depth camera (which sees how far away things are) to double-check.
Analogy: Imagine you are trying to grab a coin on the floor. You first guess the general spot. Then, you get down on your hands and knees to look closer to make sure you aren't grabbing a shadow or a crack in the floor. This ensures the robot's fingers close on solid metal, not empty air.

5. The Result: "Zero-Shot" Magic

The most impressive part is that this robot never trained on these objects.

Traditional Robots: Are like students who memorized a specific textbook. If you give them a question from a different book, they fail.
ORACLE-Grasp: Is like a genius who understands the concept of "holding." If you show it a toaster, a weird alien artifact, or a AI-generated picture of a floating shoe, it uses its general knowledge to figure out, "Ah, that part looks like a handle, I'll grab there."

Why is this a big deal?

No Manuals Needed: You don't need to teach the robot about every new object in your house.
Human-Like: It grabs things the way humans do (by the handle, not the business end).
Safe & Smart: It avoids grabbing fragile parts or dangerous edges because it understands function, not just shape.

In short, ORACLE-Grasp turns a robot from a clumsy machine that needs a manual for every object into a clever assistant that can look at a strange new thing and say, "I know exactly how to pick that up," just by using its common sense.

1. Problem Statement

Robotic grasping in unstructured, real-world environments (e.g., homes, hospitals) faces a critical challenge: generalization to novel objects without prior training.

Limitations of Traditional Methods: Existing approaches rely heavily on dense training datasets, precise 3D geometric modeling, or supervised learning. These methods often fail when encountering unseen objects or require extensive data collection.
The Semantic Gap: While deep learning has improved geometric grasp prediction, it often neglects semantic reasoning. A robot might grasp a hammer by its head (geometrically stable) rather than its handle (functionally correct for hammering).
Limitations of Current LMMs: Large Multimodal Models (LMMs) possess strong semantic understanding but lack spatial precision. They struggle to output pixel-accurate bounding boxes or specific grasp coordinates, often producing inconsistent or loosely aligned regions that are insufficient for robotic execution.

Goal: To develop a zero-shot framework that leverages the semantic reasoning of LMMs to select affordance-aligned grasps (grasps that match the object's function) for novel objects, without task-specific training or manual input.

2. Methodology: ORACLE-Grasp Framework

ORACLE-Grasp treats an LMM as a "semantic oracle" to guide grasp selection through a structured, iterative decision process. It reformulates the open-ended problem of "where to grasp" into a multiple-choice selection task over a discretized image grid.

A. Dual-Prompt Strategy

The system employs a two-stage prompting mechanism to bridge high-level semantics and low-level spatial selection:

Scene Context Prompt (SCP):
- Input: The RGB image.
- Task: The LMM generates a concise description of the object, specifically focusing on grasp-relevant features (e.g., "handle," "grip," "cylindrical section").
- Purpose: This initiates Chain-of-Thought (CoT) reasoning, establishing the functional context before spatial selection.
Grasp Region Prompt (GRP):
- Input: The RGB image discretized into an $a \times b$ grid (e.g., $3\times3$ to $9\times9$ ) and the SCP response.
- Task: The LMM selects the optimal grid cell corresponding to the best grasp area and provides a brief explanation for its choice.
- Mechanism: By forcing the model to choose from discrete tiles rather than generating free-form coordinates, the system overcomes the LMM's spatial imprecision. The required explanation ensures the selection is grounded in the functional reasoning established by the SCP.

B. Iterative Refinement & Aggregation

Multiple Iterations: The GRP is executed $K$ times with varying grid resolutions.
Candidate Aggregation: The selected grid cells are converted into rectangular masks. Overlapping masks are identified using Intersection-over-Union (IoU).
Grasp Pose Estimation:
- Position ( $\bar{p}$ ): Calculated as the centroid of all valid mask centers.
- Orientation ( $\theta$ ): Derived using Principal Component Analysis (PCA) on the set of mask centers to determine the dominant axis of the graspable region.
Orientation Refinement: If the object is rotated relative to the grid, the image is rotated to align with the grid, the process is repeated, and the result is inverse-rotated to the original frame.

C. Early Stopping Mechanism

To improve efficiency, the system monitors the convergence of mask centers. If the distance between the current centroid and any individual mask center falls below a threshold ( $\rho \times \text{image diagonal}$ ), the system triggers an early stopping protocol. It then performs additional iterations on a cropped region centered on the object to refine the selection further.

D. Depth-Based Refinement (Optional)

For RGB-D inputs, a heuristic refinement step corrects potential errors where the 2D prediction might point to empty space (e.g., the hollow center of a handle).

The system samples depth values along the line of sight from the camera to the predicted point.
It selects the closest valid surface point ( $\bar{p}^*$ ) within a clearance radius defined by the gripper's physical dimensions, ensuring the robot targets a solid, reachable surface.

3. Key Contributions

ORACLE-Grasp Framework: A novel zero-shot system that uses LMMs as semantic oracles to infer human-like, task-relevant grasps without training data, labels, or custom encoders.
Dual-Prompting Strategy: A method combining Scene Context Prompting (for functional reasoning) and Grasp Region Prompting (for spatial selection via grid discretization), enabling structured CoT reasoning directly from a single image.
Context-Free Affordance Reasoning: Demonstrates that LMMs can infer the most probable functional grasp (e.g., handle vs. head) for unfamiliar objects without explicit user grounding or task instructions.
Integration of Depth: A mechanism to optionally integrate depth data for metric refinement, bridging the gap between high-level semantic reasoning and precise physical execution.
Empirical Validation: Extensive testing on diverse real-world and AI-generated objects, showing high success rates in zero-shot robotic pick-up tasks.

4. Experimental Results

The framework was evaluated using LLaMa 3.2-Vision (11B parameters) on two test sets:

Test Set I (RGB): 100 images of diverse objects (including AI-generated ones).
Test Set II (RGB-D): 20 real-world objects tested on a Franka Research 3 robot arm.

Key Findings:

Accuracy: ORACLE-Grasp significantly outperformed baselines (direct bounding box generation).
- Position Error: Achieved a Normalized Root Mean Squared Error (NRMSE) of 5.1 ± 4.2 (with SCP+GRP) compared to 61 for the baseline.
- Orientation Error: Mean Absolute Error (MAE) of 19.0° ± 14.8°.
Robotic Success Rate: In real-world experiments, the system achieved an 88% average pick-up success rate across 20 novel objects.
Impact of Components:
- Depth Refinement: Reduced position RMSE by over 70% for objects like coffee bags and cardboard boxes.
- CoT Reasoning: The "Brief Explanation" requirement in the GRP significantly improved spatial precision by forcing the model to justify its choice.
- Efficiency: The early stopping mechanism reduced inference time by an average of 27.48% in 89% of runs.
Failure Cases: Performance was lower on objects with ambiguous grasp regions (e.g., cardboard boxes) or large objects with small handles (e.g., briefcases), suggesting a need for more iterations in complex scenarios.

5. Significance and Future Work

Significance:

Paradigm Shift: Moves robotic grasping away from "expert students" (supervised models trained on specific datasets) toward "reasoning generalists" (LMMs that can handle any object).
Affordance Alignment: Prioritizes functional suitability (grasping a drill by the handle) over mere geometric stability, a critical step for service robots interacting with humans.
Accessibility: Uses open-weight models and standard RGB/RGB-D cameras, avoiding the need for proprietary APIs or complex 3D reconstruction pipelines.

Limitations & Future Directions:

Inference Latency: LMM inference is currently too slow for high-frequency real-time control loops; future work may involve model distillation or compression.
Single View: The system operates from a single viewpoint. Future iterations could incorporate active exploration or multi-view fusion to handle occlusions.
Multi-Object Scenes: Current validation is limited to single-object scenes. Future work aims to extend the framework to cluttered environments by parameterizing the SCP for specific target identification.

In conclusion, ORACLE-Grasp demonstrates that Large Multimodal Models can serve as powerful, zero-shot semantic oracles for robotic manipulation, enabling flexible and human-like grasping strategies in dynamic, unstructured environments.