PROBE: Manipulation-Grounded Visual Question Answering with VLM Agents
This paper introduces PROBE, a framework comprising a high-fidelity simulator (PROBE-Sim), a benchmark (PROBE-Bench), and a fine-tuning recipe (PROBE-Agent) to advance Vision-Language Model agents in Manipulation-Grounded Visual Question Answering, demonstrating that agentic tool-based manipulation significantly outperforms perception-only approaches in dynamic, cluttered environments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a robot standing in a busy kitchen, asked a simple question: "Is my medication still in the cabinet?" In a perfect, empty world, the robot could simply look at the cabinet door and answer. But in the real world, the cabinet is cluttered. A row of cereal boxes, a jar of coffee, and a stack of plates might be hiding the medicine bottle completely. To answer correctly, the robot cannot just look; it must reach out, move the boxes, and peek inside. This is the gap between seeing a picture and understanding a world. Current artificial intelligence systems are excellent at analyzing static images, where the answer is right there on the screen. However, they struggle when the answer requires physical action to reveal hidden information. They do not yet know how to decide when to stop looking and start moving.
Researchers at NVIDIA have built a new way to teach robots this specific skill, bridging the gap between seeing and doing. They created a system called PROBE, which stands for a framework for testing and training robots to answer questions by physically manipulating their surroundings. The team realized that for a robot to be truly helpful in a home, it must be able to handle clutter. If a question cannot be answered from the first view, the robot needs to know how to clear a path, move an object, or lift a lid to find the truth. To study this, the researchers did not just write a theory; they built a high-fidelity digital world filled with nearly 1,800 real-world objects, from cups and bowls to tools and containers. In this simulation, a robot arm can reach in, grab items, push them aside, or lift them up, all while a camera watches from above.
The researchers used this digital playground to create a rigorous test called PROBE-Bench. This test consists of 150 different scenarios where a robot is asked questions about a messy table. The questions range from simple counts, like "How many red cups are there?", to complex inquiries about what is hidden underneath other items. The key challenge is that the robot cannot answer these questions just by looking at a snapshot. It must decide if it needs to move something first. The team tested eight different advanced artificial intelligence models on this task. Some models were allowed to just look at the image and guess, while others were given a set of tools to physically interact with the scene. The results were clear: when the models were allowed to use their tools to move objects, they got significantly better at answering the questions. On average, giving the robot the ability to act improved its success rate by eight percentage points. The improvement was even more dramatic for some of the less powerful models, which saw their accuracy jump by as much as sixteen percentage points once they could move things around.
However, the researchers found that having the tools was not enough; the robot also needed to know when to use them. In some tasks, the robot would correctly identify that it needed to move an object, but in others, it would move the wrong thing or move things unnecessarily, leading to errors. The most difficult tasks were those where the robot had to figure out exactly which object to move without being told. To solve this, the team developed a training method called PROBE-Agent. They took a very powerful, large artificial intelligence model that was already good at these tasks and used it as a teacher. This teacher model demonstrated the correct sequence of actions: looking, deciding to move an object, moving it, and then answering. The researchers then taught a smaller, more efficient model to mimic these successful actions.
This teaching process worked remarkably well. The smaller models, after being trained on the teacher's successful moves, became much better at the task. They learned not just to use the tools, but to use them efficiently. In fact, the trained smaller models performed almost as well as the powerful teacher model itself. They also showed a ability to generalize, meaning they could apply what they learned to new objects and new types of questions they had never seen before. For example, when asked to find what was hidden under a specific cover, the trained models could figure out the right sequence of moves even if the cover was a new type of object. The team then took these trained models and tested them on a real robot in the physical world. Even though the robot was trained entirely in a digital simulation with a camera looking straight down, and the real robot had a camera looking from the front, the trained model still performed very well. It successfully navigated real cluttered tables, moved real objects, and answered questions with a high degree of accuracy, proving that the skills learned in the simulation could transfer to reality.
The work highlights a crucial step forward for robotics. It shows that for robots to be useful in our homes, they must move beyond passive observation. They need to be active participants in their environment, capable of changing the scene to find the answers they seek. While the current systems are not perfect and still struggle with the most complex multi-step tasks, the research demonstrates that teaching robots to combine vision with physical action is a viable path forward. By using simulations to train models on how to manipulate their world, researchers are building a foundation for robots that can truly understand and interact with the messy, cluttered reality of human life. The study suggests that the future of helpful robots lies not just in seeing better, but in acting with purpose.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.