Hybrid Visual Perception and LLM Framework for Service Robot Planning and Navigation

Service robots are no longer just a concept for the future; they are beginning to appear in hospitals, hotels, and public spaces, tasked with guiding people and performing simple chores. For these machines to be truly useful, they must do more than just follow a pre-programmed path. They need to understand what a human is asking them to do, see the world around them to avoid obstacles or find specific items, and then figure out how to move to get the job done. The challenge has long been how to combine these three abilities—understanding language, seeing the environment, and planning movement—into a single system that is smart enough to be helpful but small and fast enough to run on a robot's own onboard computer.

For years, researchers have tried to solve this by building massive, all-in-one artificial intelligence systems that try to do everything at once. These systems, often called vision-language models, are incredibly powerful because they can look at a picture and read a sentence simultaneously. However, they are also incredibly heavy, requiring huge amounts of computing power that simply does not fit inside a small, battery-operated robot. This creates a dilemma: a robot that is smart enough to understand complex instructions but too slow to react in real time, or a robot that is fast but too dumb to understand what it is seeing.

A team of researchers from Indonesia, Japan, and Belgium has proposed a different way forward. Instead of forcing one giant brain to do everything, they built a system that splits the work into two specialized parts that talk to each other. In their new framework, a fast, lightweight visual system handles the job of spotting objects, while a separate language model handles the job of understanding instructions and making plans. This separation allows the robot to run on a small, embedded computer without needing a massive external server. The researchers tested this approach on a guide robot designed to help people navigate public buildings, and they found that by keeping the tasks separate, the robot could understand complex commands, find objects, and move to specific locations with high accuracy, all while reacting much faster than the all-in-one systems.

The robot they used was built on a standard mobile base but was equipped with a single camera and a compact computer chip known as the Jetson Orin Nano. This chip is powerful enough to run modern software but small enough to fit on a robot that needs to move freely without being tethered to a wall outlet. The researchers programmed the system to accept natural language commands from a human, such as "Go to the front office and find a person." When the robot receives such a command, the language part of the system breaks it down into a list of steps. It decides that the robot needs to move to a specific location and then look for a person. Crucially, it tells the visual part of the system exactly what to look for, rather than asking the visual system to guess what might be important.

This division of labor is the key to the system's speed. In traditional all-in-one systems, the computer has to process the entire image and the entire sentence at the same time, which takes a long time. In this new setup, the language model quickly decides that the robot needs to look for a "person," and then it sends a simple, short instruction to the visual detector to find only that specific object. The visual detector, which is designed to be very fast, scans the camera feed and reports back if it sees a person. Because the visual system isn't wasting time analyzing every single thing in the room, and the language system isn't wasting time trying to interpret the pixels, the whole process happens much faster. The researchers measured this speed difference and found that their split system was between three and twenty times faster than the all-in-one systems, depending on whether the computer was running locally or connecting to a cloud server.

To see if this approach actually worked in the real world, the team ran a series of tests. First, they tested the robot's ability to follow movement commands alone, asking it to go to the nearest room or the front office. In these tests, the robot successfully planned its route and arrived at the correct destination 96 percent of the time. Next, they added the visual component, asking the robot to go to a location and then find a specific object, like a fire hydrant, a chair, or a person. In these more complex scenarios, where the robot had to navigate and then search, the system succeeded 84 percent of the time. The few failures that did occur were usually because the robot's single camera had trouble judging exactly how close it was to an object, causing it to stop slightly too early or too late, rather than because it misunderstood the command or failed to see the object.

The researchers also took the system out of the computer simulation and placed it on the actual physical robot in a real building lobby. They gave it the same types of commands, and the robot performed the tasks reliably, navigating through the space, stopping to look for people, and speaking back to the user. This real-world test confirmed that the system could handle the messiness of a real environment, including people walking by and changing lighting conditions, without needing any structural changes to the software. The results showed that the robot could interpret a human's voice, figure out where to go, find what was needed, and execute the task, all while running on a small computer that fits on the robot itself.

The study suggests that the future of service robots may not rely on making the robots' brains bigger and more complex, but rather on organizing how they think. By separating the job of seeing from the job of reasoning, the researchers demonstrated that a robot can be both highly capable and efficient. While the all-in-one systems are impressive, they are often too heavy for practical use on mobile robots. This hybrid approach offers a path forward where robots can be deployed in hospitals, hotels, and offices to assist people, reacting quickly to instructions and moving safely through crowded spaces. The work does not claim to have solved every problem in robotics, but it provides a clear and proven method for making robots that are smart enough to help and fast enough to be useful.