RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision
RoboFind is a multi-agent framework that enables blind and low-vision users to locate specific personal objects by combining a smartphone-based teaching interface with a quadruped robot's autonomous search and verification capabilities, achieving significantly higher success and reliability rates than baseline methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
For people who are blind or have low vision, the world is often navigated through sound, touch, and memory. While technology has long offered tools to help with movement, such as guide dogs or canes, a different kind of challenge remains: finding a specific object in a room. Current solutions often rely on the user walking around, holding a camera, and hoping to catch a glimpse of what they are looking for. If the object is hidden behind a chair or tucked in a corner, the camera cannot see it, and the search fails. The question researchers have been asking is whether a robot could take over the physical work of the search, moving through a space to find an item while the user stays safely in place, all without needing to see the result themselves.
This is the core problem addressed by a new system called RoboFind. The researchers, working at the Karlsruhe Institute of Technology in Germany, built a framework that connects a smartphone, which the user holds, with a four-legged robot that does the searching. The system is designed to handle a very specific difficulty: finding a particular item, like a favorite blue backpack or a specific pair of glasses, rather than just any object of that type. A robot that can only find "a backpack" might bring back the wrong one, leaving the user frustrated. RoboFind solves this by separating the task into two distinct phases: teaching the robot what to look for, and then verifying that it has found the exact right thing before telling the user the job is done.
The process begins with the user teaching the robot. Using a smartphone app designed to be accessible through voice and touch, the user records short videos of the object they want to find. The app guides the user to move the phone around the item, capturing it from the front, side, and top, as well as against a different background. This is not just a simple photo; the system analyzes these videos to create a detailed digital profile of the object. It stores a collection of visual references that describe exactly what that specific backpack looks like, distinguishing it from any other bag in the house. Once this profile is saved, the user can select it from a list and ask the robot to find it.
When the search begins, a quadruped robot, similar in shape to a dog, enters the room. It does not know where the object is, so it uses a navigation system to explore the environment, moving forward, turning, and scanning the space. When the robot's navigation system thinks it has found a potential match, it stops. However, this is where the system differs from earlier attempts. Instead of immediately announcing that the object has been found, the robot pauses and sends a picture of what it sees back to the smartphone for a second check. A separate part of the software compares this new picture against the original reference bank created during the teaching phase. It asks a simple question: does this look exactly like the specific item the user taught me?
If the picture matches the stored references closely enough, the robot confirms the find and reports success to the user. If the picture does not match—for instance, if the robot stopped in front of a different black bag that looks similar but is not the right one—the system rejects the candidate. The robot does not give up; it clears its memory of that spot, turns around, and continues searching until it finds the correct item. This loop of searching, stopping, checking, and either accepting or continuing is the heart of the system. It ensures that the robot does not simply guess based on a general description but verifies the identity of the object before declaring the mission complete.
The researchers tested this system in real-world settings, including bedrooms, living rooms, kitchens, and even gardens. They ran thirty-two separate missions with ten different target objects, ranging from movable items like umbrellas and towels to stationary objects like chairs and toilets. In these trials, the system succeeded in finding the correct object in 85 percent of the attempts. To understand how effective this was, the researchers compared it to a simpler approach where the robot would stop at the first object that looked somewhat right and declare victory. That simpler method succeeded only 25 percent of the time and was wrong three-quarters of the time, often leaving the user with the wrong item. The new system also outperformed a version that relied solely on a powerful artificial intelligence model without the specific verification step, which succeeded in less than half of the shared trials.
The data showed that the extra time spent checking and re-searching was worth it. While the successful missions took an average of about three minutes and forty-five seconds, the system rarely made a mistake. In contrast, the simpler methods were faster but frequently led to false successes, where the robot claimed to have found the object when it had not. The verification step was crucial; it filtered out nearly all the incorrect stops. When the robot did stop at the wrong place, the system caught the error, sent the robot back out, and eventually found the right object. This ability to recover from a mistake and keep searching until the exact match was found is what makes the system reliable for a user who cannot visually confirm the result.
The study also looked at how the system handled different types of objects. For items that can be moved, like a cup or a backpack, the system relied on the visual appearance of the object itself. For items that stay in one place, like a specific chair at a desk, the system also checked the surroundings to ensure the object was in the expected context. This distinction allowed the robot to handle the complexity of a real home, where many similar objects might exist. The researchers noted that while the system is highly effective, it is not perfect; in a few rare cases, the robot struggled to distinguish between very similar items or encountered technical interruptions, but these were exceptions rather than the rule.
Ultimately, this work demonstrates that a robot can be a trustworthy partner for people with visual impairments, not just by moving them from place to place, but by actively searching for their personal belongings. The key innovation is the separation of the search from the confirmation. By letting the robot do the heavy lifting of moving through the space and then using a rigorous check to ensure the right object is found, the system bridges the gap between what the robot sees and what the user needs. The results suggest that with this kind of multi-step verification, robots can move beyond simple navigation to become reliable assistants for everyday tasks, giving users the confidence that when the robot says it has found their item, it truly has.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.