⚡ electrical engineering

Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning

The paper introduces GRASP, a neuro-symbolic planning framework that leverages pretrained Vision-Language Models to translate natural language into grounded bounding-box goals, enabling zero-shot open-vocabulary tabletop manipulation with a 73.3% success rate on real robots without task-specific training.

Original authors: Allison Andreyev, Landon Eum, Nestor Tiglao, Romel Gomez

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Allison Andreyev, Landon Eum, Nestor Tiglao, Romel Gomez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot arm sitting on a table, and you want it to pick up specific items and put them in specific places. Usually, programming a robot is like teaching a dog complex tricks: you have to show it exactly what to do thousands of times, or you have to give it very rigid, boring instructions like "move 5 inches left, then 2 inches up." If you say something new, like "put the red things on the top shelf," the robot often gets confused because it hasn't seen that exact command before.

This paper introduces a new system called GRASP (Grounded Reasoning and Symbolic Planning) that acts like a smart translator and a GPS for the robot. It allows the robot to understand natural language and figure out how to move without needing to be retrained for every new task.

Here is how it works, broken down into simple steps:

1. The "Translator" (The Brain)

When you speak to the robot, say, "Put all the blue objects on the top shelf," the system doesn't just hear words; it translates them into a map.

The Analogy: Think of a human giving a tourist directions. Instead of saying "Go to coordinate X, Y," the human says, "Find the blue building and go to the top floor."
How GRASP does it: It uses a powerful AI (called a Large Language Model) to turn your sentence into a list of "goals." It figures out what "blue objects" are and defines the "top shelf" as a specific area on a screen. It creates a digital checklist of what needs to be done.

2. The "Eyes" (The Vision)

Once the robot knows the goal, it needs to find the objects.

The Analogy: Imagine you are looking for your keys in a messy room. You don't scan every single item randomly; you look for the shape and color of your keys.
How GRASP does it: It uses a specialized "eye" (a computer vision model called GroundingDINO) that is already trained to recognize thousands of things. It scans the table and draws invisible boxes around the objects that match your description (e.g., it draws a box around the blue bottle).

3. The "Steering Wheel" (The Control)

This is the most unique part. Instead of the robot trying to memorize a complex path to grab the object, it uses a simple, continuous feedback loop.

The Analogy: Think of playing the game "Hot and Cold" or aiming a camera at a moving target. You don't calculate the exact math to hit the target in one shot. Instead, you look at where the target is, move a little bit toward it, look again, and move a little more. You keep adjusting until you are right on top of it.
How GRASP does it: The robot looks at the "box" around the object on its screen. If the box is to the left of the center of its view, the robot moves its arm slightly left. If the box is too small (meaning the object is far away), it moves closer. It does this over and over again, constantly correcting its path, until the object is perfectly centered in its "gripper."

Why is this special?

Most advanced robot systems are like heavyweight athletes that need years of training (thousands of practice attempts) to learn a new task. They are also very slow and expensive to run.

GRASP is like a lightweight, adaptable hiker.

No Training Needed: It doesn't need to practice the specific task. If you ask it to pick up a "magenta marker" or a "lime green scissor," it figures it out on the spot using its pre-trained "eyes" and "translator."
It's Robust: Because it keeps checking its position (the "Hot and Cold" loop), it can handle it if the object moves slightly or if the camera angle isn't perfect. It doesn't just guess and hope; it corrects itself in real-time.

The Results

The researchers tested this system with different levels of difficulty, from easy tasks (picking up a big block) to hard tasks (picking up small, tricky items like a screwdriver).

Success Rate: The robot successfully grabbed the right items about 73% of the time overall.
Where it failed: It mostly struggled when the "eyes" couldn't see the object clearly (like if it was too far away or the lighting was bad), not because the robot's "brain" gave it the wrong instructions.

In Summary

GRASP is a system that lets you talk to a robot like a human, and the robot uses a combination of AI translation and continuous visual correction to figure out how to grab things. It skips the need for heavy training and rigid programming, making it a much more flexible tool for everyday tasks like sorting items on a table.

Technical Summary: GRASP – Language-Conditioned Grasping via Neuro-Symbolic Planning

Problem Statement
The integration of robotics into dynamic household and industrial environments requires machines to adapt to natural-language prompts in real time. While Vision-Language Models (VLMs) and Large Language Models (LLMs) have enabled zero-shot generalization in Task and Motion Planning (TAMP), current state-of-the-art approaches face significant limitations. Many rely on computationally "heavyweight" models requiring extensive training on thousands of demonstrations, or they depend on rigid symbolic structures (e.g., fixed color lists or hard-coded coordinates) that fail to capture abstract spatial concepts like "top shelf." Furthermore, existing pipelines often entangle high-level reasoning with low-level control, creating bottlenecks in efficiency and modularity. There is a critical need for a framework that bridges flexible language understanding with robust, interpretable execution without requiring task-specific fine-tuning or policy learning.

Methodology: The GRASP Framework
The authors introduce GRASP (Grounded Reasoning and Symbolic Planning), a lightweight neuro-symbolic framework designed for open-vocabulary tabletop manipulation. GRASP decouples high-level reasoning from low-level control through a two-component architecture:

Neural Component (Perception and Reasoning):
- Goal State Generation: An LLM (specifically GPT-5.2 in the experiments) parses natural language instructions to generate an explicit symbolic goal state. This involves extracting target object queries (e.g., "blue objects") and mapping spatial constraints (e.g., "top shelf") into image-coordinate thresholds (e.g., $y \leq \tau_1$ ). The output is a JSON representation of desired bounding boxes.
- Object Detection: A pretrained VLM, GroundingDINO (G.DINO), is used to detect objects in the scene based on the labels extracted by the LLM. G.DINO is selected for its ability to limit detections to a defined set of labels and its strong performance against other SOTA VLMs. It provides a continuous flow of bounding box detections from both a global shelf camera and an end-effector camera.
Symbolic Component (Planning and Control):
- Goal Similarity Evaluation: The system computes a normalized similarity score ( $S$ ) between the LLM-generated goal state and real-time detections. This score combines Intersection over Union (IoU) and the Euclidean distance between box centers. If the similarity exceeds a threshold or no objects are detected for consecutive frames, the task terminates.
- Closed-Loop Motion Control: The system employs a discrete-time closed-loop control pipeline. It calculates the error between the detected bounding box center and the camera's optical center. A proportional Roll-Pitch-Yaw (RPY) controller adjusts the robot's end-effector based on this error.
- Stabilization Mechanisms: To ensure stability, the controller utilizes exponential smoothing on the error signals and a deadband to suppress jitter. The system selects the target object based on the highest logit (confidence) from the detector.

Key Contributions
The paper outlines three primary contributions:

GRASP Framework: A neuro-symbolic system that compiles natural language instructions into explicit symbolic goal states, grounding them via a pretrained VLM without additional fine-tuning.
Zero-Shot Execution: Demonstration that closed-loop goal evaluation enables task execution without policy learning or reinforcement learning.
Lightweight Pipeline: A modular architecture linking open-vocabulary detection to continuous motion via proportional control, avoiding the computational overhead of end-to-end learning.

Experimental Results
The authors evaluated GRASP across 90 trials spanning three difficulty levels (Easy, Medium, Hard) involving various objects (e.g., blocks, tape, tools).

Success Rates: The system achieved success rates of 86.67% (Easy), 76.67% (Medium), and 56.67% (Hard), resulting in an overall success rate of 73.33%.
Failure Analysis: Performance degradation in harder tasks was attributed primarily to missed or incorrect G.DINO detections and hardware limitations (limited Field of View and reduced image quality at distance), rather than failures in the control pipeline itself.
Ablation Studies:
- Closed-Loop vs. Open-Loop: Removing closed-loop feedback (running adjustment only once) significantly reduced success rates, confirming the necessity of iterative adjustment.
- Smoothing & Deadband: Removing these components degraded performance, indicating that unconstrained corrections introduce instability.
- Target Selection: Selecting targets based on random logits or first-match strategies performed worse than selecting the highest-confidence (logit) detection.

Significance and Claims
The paper positions GRASP as a step toward scalable, general-purpose robotic sorting and arrangement. The authors claim that by decoupling reasoning from control, GRASP offers an interpretable and efficient solution that eliminates the need for extensive training data or policy learning. The system successfully interprets abstract spatial concepts and executes tasks in open-vocabulary settings. The authors modestly note that while the current work focuses on grasping and alignment, full end-to-end sorting evaluation (including placement verification) is reserved for future work due to hardware constraints. The framework is presented as a viable alternative to heavy, training-dependent models for dynamic, language-conditioned manipulation.