🤖 machine learning

Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Prediction

This paper identifies part grounding, rather than a lack of action knowledge, as the primary bottleneck in Vision-Language Model affordance prediction, demonstrating that explicitly specifying the target object part dramatically improves action accuracy across all tested models.

Original authors: Sarthak Sattigeri

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sarthak Sattigeri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots are getting better at seeing the world. They can identify a chair, a cup, or a camera just by looking at a photograph. But there is a gap between seeing an object and knowing how to move it. To make a robot useful, we need it to understand "affordances," a concept that simply means knowing what an object is for and how a human or machine should interact with it. If a robot sees a drawer, it must know to pull it. If it sees a button, it must know to push it. This seems obvious to a human, but for artificial intelligence, it is a surprisingly difficult puzzle. Researchers have been testing vision-language models—systems that can look at an image and answer questions about it—to see if they can figure out these interactions. The results have been disappointing, with the machines often failing to give the right instructions. The big question has been why: do these models lack the basic knowledge of how things work, or are they just looking at the wrong part of the picture?

A researcher set out to solve this mystery by testing eight different artificial intelligence models on a specific task involving nineteen different objects, such as cameras, drawers, and lids. They asked the models a simple question: given a picture of an object, what motion should a robot perform on it? The correct answers were limited to a small set of actions, like pushing, pulling, or lifting. When the models were allowed to choose which part of the object to talk about, they performed very poorly. In fact, when the correct answer was to "push" something, the models almost never said that word. Out of sixty-four times where pushing was the right move, the models only got it right once. The rest of the time, they suggested actions that were completely wrong for the situation.

At first glance, this looked like a major failure of knowledge. It seemed the models simply did not know that buttons are pushed and drawers are pulled. However, when the researcher looked closer at what the models were actually saying, they found a different story. The models were not confused about the action; they were confused about the target. When shown a camera and asked what to do, the models often described how to pick up the entire camera body, rather than explaining how to press the button on the back. They were answering a reasonable question about the object as a whole, but they were missing the specific part that needed to be acted upon. The models were looking at the wrong piece of the puzzle.

To test this theory, the researcher changed the experiment. Instead of letting the models choose which part to discuss, they told the models exactly which part to focus on. They said, "Look at the button on this camera. What should a robot do with it?" The change was dramatic. As soon as the researcher named the specific part, the models' accuracy jumped significantly. Every single model improved, with their success rates rising from a low of about fifteen percent to a high of nearly ninety-five percent. The models that had previously failed to suggest "push" even once suddenly got it right almost every time. They also started using the correct language, describing how a button moves inward when pressed, even when they were not given a list of words to choose from.

This finding suggests that the problem was not a lack of physical knowledge, but a failure to ground the question in the right location. The models knew what a button was for; they just could not reliably find the button in the image on their own. The researcher also tested the models' ability to point to the exact spot on the object where a robot should grab it. On real photographs, some models did okay, but on computer-generated images, none of them could point better than a random guess. This confirmed that the weak link was indeed the ability to locate the specific part of an object that matters.

The study also uncovered a few mistakes in how the researcher had set up their own tests. They realized that their initial way of measuring success was too easy in some ways, allowing a model to score high just by guessing the center of the image, and too strict in others, penalizing models for minor errors in how they defined an action. Once these measurement errors were fixed, the results became even clearer. The models were not missing the rules of physics; they were missing the ability to focus their attention on the right detail.

The takeaway for anyone building robots is practical and specific. If you want a robot to manipulate an object, you cannot just hand it a picture and ask what to do. The system needs a way to first identify the specific part of the object that needs attention. Once that part is named, the artificial intelligence can reliably tell you how to move it. The models are not broken; they just need a little help to know where to look. This distinction changes how we think about the future of robotics. Instead of trying to teach machines every possible rule of mechanics, we might just need to give them better tools to find the right part of the world to act upon. The knowledge is there; it just needs to be pointed in the right direction.

Technical Summary: Part Grounding, Not Action Knowledge

Problem Statement

Current benchmarks indicate that Vision-Language Models (VLMs) perform poorly on low-level manipulation tasks, specifically regarding affordance prediction (determining how to interact with objects). However, aggregate accuracy metrics fail to distinguish between two distinct failure modes:

Action Knowledge Deficit: The model lacks the physical understanding of what action a specific object part requires (e.g., knowing a button must be pushed).
Part Grounding Failure: The model possesses the necessary action knowledge but fails to identify which part of the object is the relevant target for interaction.

The paper posits that existing evaluations conflate these steps, leading to the assumption that VLMs lack physical reasoning capabilities when the bottleneck may actually be spatial attention or part identification.

Methodology

The study evaluates eight VLMs from three independent developers (Alibaba, OpenAI, and Anthropic) across three distinct prompting conditions using 19 articulated objects (from GAPartNet/PartNet-Mobility) and 28 real tool photographs.

The action labels are deterministically mapped from annotated part categories (e.g., slider button $\rightarrow$ "push", hinge door $\rightarrow$ "pull"). The evaluation compares three prompt types:

Prompt A (Standard/Open): The model must identify the relevant part and select the correct action from a constrained label set.
Prompt C (Grounded): The target part is explicitly named in the prompt; the model must select the correct action from the same label set.
Prompt B (Free Prose): The target part is named; the model describes the action in free text without a constrained label set.

This design isolates part selection (A vs. C) from action knowledge (C vs. B). Additionally, the study includes a spatial localization task where models must predict grasp points, comparing their performance against a constant baseline that ignores the image.

Key Results

1. The "Push" Blind Spot and Part Grounding

Under Prompt A, models exhibited a severe failure to predict the "push" action, despite it being the correct answer for 8 of the 19 objects and present in the label set.

Observation: "Push" was generated only once across 64 evaluations (by gpt-5). Seven of the eight models never produced it.
Analysis: Inspection revealed models were not failing to know how to push; they were failing to identify what to push. Models described interacting with irrelevant parts (e.g., the "camera body" or "lid") rather than the specific mechanism (e.g., the "button").
Impact of Grounding: When the target part was named (Prompt C), action accuracy improved significantly for all models, ranging from +0.32 to +0.63.
- "Push" recall jumped from 0/8 (or 1/8) to 7/8 or 8/8.
- All eight models exceeded the performance of a constant baseline (which ignores the image) under Prompt C, whereas none did under Prompt A.

2. Action Knowledge is Present

Under Prompt B (free prose, named part), models successfully generated "pushing" language for 6 to 8 of the 8 relevant objects. This confirms that the models possess the semantic vocabulary and conceptual understanding of the action; the failure in Prompt A was strictly due to the inability to ground the action to the correct visual part.

3. Spatial Localization Weakness

Spatial localization results mirrored the action findings:

Real Photographs: Only three models localized grasp points better than a constant baseline.
Rendered Objects: No model outperformed the constant baseline in localization.
Since the action task used rendered objects, the poor localization performance supports the hypothesis that part grounding is the primary bottleneck.

4. Measurement Corrections

The author identified and corrected two measurement errors that initially skewed results:

Spatial Threshold: The hit-rate threshold (0.15 of image diagonal) was too loose for centered objects, allowing a constant baseline to score 0.929.
Labeling Rules: An initial geometric rule for mapping parts to actions was incorrect for 4 of the 19 objects (misclassifying buttons and drawers). Correcting these errors actually increased model accuracy, indicating the models were previously penalized for correct reasoning based on flawed ground truth.

Key Contributions

Decoupling Grounding from Knowledge: The paper demonstrates that the low performance of VLMs in manipulation benchmarks is driven primarily by part grounding (identifying the target) rather than a lack of action knowledge (knowing the physics).
Prompt Sensitivity: It shows that providing the grounding variable (naming the part) allows all tested models to surpass a constant baseline, suggesting they are more capable than open-prompt evaluations indicate.
Benchmark Critique: The study highlights that current affordance evaluations often measure part grounding capabilities while attributing failures to physical reasoning deficits.
Methodological Rigor: The paper documents how trivial baselines and labeling errors can produce misleadingly low performance metrics, emphasizing the need for careful audit of evaluation pipelines.

Significance and Claims

The paper concludes that the dominant bottleneck in VLM affordance prediction is part grounding, not the absence of a general model of mechanics.

Practical Implication: For robotics pipelines, these models are more useful than previously thought, provided a separate module (e.g., a part detector) handles the initial grounding. A pipeline that segments parts first and then queries the VLM performs substantially different and more effective work than one that hands the VLM a raw image and a goal.
Modest Scope: The author explicitly states they do not claim the models possess a general theory of mechanics. Condition C (naming the part) supplies the grounding variable, and the task reduces to mapping a known part category to a conventional action. The results establish that the models can perform this mapping if the visual grounding is supplied, but they do not prove the models can derive the grounding themselves.

The findings suggest that future research and benchmarking should distinguish between part identification and action selection to avoid misdiagnosing VLM capabilities in physical reasoning tasks.