Imagine a robot arm reaching for a coffee mug. To succeed, the machine must solve three distinct puzzles at once: it must decide which part of the mug matters (the handle, not the rim), locate exactly where that part is in the visual world, and understand what action that part invites (grasping, not pushing). For years, researchers have tested artificial intelligence systems on these tasks by asking them to describe what a robot should do with an object. A recent companion study suggested that if you simply tell the AI the name of the target part—saying "grab the handle" instead of just "grab the mug"—the system's performance jumps dramatically. This led to a hopeful conclusion: the AI was finally learning to look at the image and reason about the object's physical mechanics.

However, a new investigation called GroundBench suggests this conclusion might be premature. The researchers behind this study, led by Sattigeri at Manipal University Jaipur, suspected that the AI was not actually learning to see the object better. Instead, they hypothesized that the system might be relying on a shortcut: simply memorizing that certain words, like "handle," are almost always paired with the action "grab," regardless of what the image actually shows. To test this, they built a rigorous new benchmark designed to separate the ability to find a part in an image from the ability to guess an action based on a word. They did not just ask the AI to perform a task; they systematically stripped away information, one piece at a time, to see which specific clue was actually driving the success.

The researchers constructed a series of six different scenarios, or conditions, to test three different versions of a leading artificial intelligence model. In the first scenario, the AI saw only the image of an object, like a keyboard or a door, with no text instructions. In the second, they added the name of the object, such as "keyboard." In the third, they named the specific part, like "the spacebar." In the fourth, they provided the exact location of the part on the screen—a specific point and a box around it—but deliberately withheld the name of the part. In the fifth, they gave both the name and the location. Finally, in the sixth, they added technical details about how the object moves, such as whether a joint is a hinge or a slider.

The results were striking and counterintuitive. When the researchers gave the AI the exact location of the target part but refused to tell it what the part was called, the system's performance collapsed. Across the three models tested, the accuracy for choosing the correct action dropped to the level of a random guess, or even lower. One model, when given the location of a button but not its name, scored only 0.26, while a simple baseline that ignored the image entirely scored 0.53. The AI could perfectly reproduce the location it was shown, placing its "finger" exactly where the researchers pointed, yet it failed to understand what to do with that location. It was as if the robot could see the button but had no idea that buttons are meant to be pushed.

In sharp contrast, when the researchers gave the AI the name of the part but withheld its location, the performance soared. The same models that failed with location-only data jumped to accuracy rates between 0.68 and 0.74 when they were told the part was a "button" or a "door," even without seeing where it was. This pattern held true across the board: the moment the AI was given the category name of the part, it could almost always guess the correct action. The researchers found that in their carefully curated set of objects, the name of the part alone was enough to determine the answer. The AI was not looking at the image to figure out the physics; it was reading the word and recalling a pre-learned association.

To confirm this suspicion, the researchers ran a control test where they removed the image entirely and asked the models to answer based on text alone. For the most advanced model tested, removing the image made no difference to its performance on the conditions where the part name was provided. The model scored just as well, or even slightly better, without seeing the object. This provided strong evidence that the system was not using visual grounding—the process of connecting words to specific pixels in an image—but was instead relying on a text-based shortcut. The AI knew that "hinge door" implies "pull" and "slider button" implies "push" because it had seen those pairings in its training data, not because it understood the mechanics of the specific door or button in front of it.

The study also tested whether the AI could follow a trick question. The researchers took an image of an object with multiple parts, such as a keyboard with many keys, and asked the AI to perform an action on a non-standard part, like a specific key that was rarely used. They wanted to see if the AI would actually look at that specific key or if it would default to the most common action for the whole object. While the models generally followed the new instruction, they struggled significantly when the requested action was unusual, such as lifting a part vertically. In these cases, the models often reverted to the standard action, suggesting they were still leaning on their most common associations rather than truly analyzing the visual scene.

Perhaps the most surprising finding was that adding more information did not always help. When the researchers gave the AI the location and the name of the part, and then added detailed mechanical descriptions about how the object moved, the performance actually went down. The models did not improve; they became less accurate. This suggests that the extra information confused the system or distracted it from the simple, effective shortcut of just using the part's name. The researchers concluded that for these specific tasks, more data did not equal better reasoning.

The implications of this work are significant for the field of robotics and artificial intelligence. It reveals that high scores on certain tests might be misleading. An AI can appear to be a master of physical reasoning when it is actually just a master of word associations. The researchers found that the dramatic improvement seen in earlier studies, where naming a part boosted accuracy by a large margin, was likely not due to the AI suddenly learning to see better. Instead, it was because the name itself contained the answer. The study does not claim that these models are incapable of visual reasoning, but it does show that under these specific conditions, they were not using it.

GroundBench serves as a diagnostic tool, a way to peel back the layers of an AI's performance to see what is actually happening underneath. By separating the visual location from the semantic name, the researchers exposed a gap between what the models seemed to be doing and what they were actually doing. They found that when the name of the part was removed, the models could not find the action, even when the location was perfectly clear. This suggests that for these systems, the path to true mechanical reasoning is longer than previously thought. They have not yet learned to look at an object and understand its function; they have learned to listen to a word and guess the function. The study ends not with a declaration of failure, but with a clearer map of where the work needs to go: building systems that can truly connect the words they hear with the physical world they see, rather than relying on the shortcuts that have served them so well until now.