HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System
HiMemVLN addresses the "Navigation Amnesia" issue in open-source zero-shot vision-and-language navigation by introducing a Hierarchical Memory System that significantly enhances long-term localization and visual recall, achieving nearly double the performance of existing state-of-the-art open-source methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a robot that can understand your voice and see the world just like you do. This is the dream of "Embodied AI," a field where computers don't just chat or calculate; they physically move through rooms, follow instructions like "go to the kitchen and grab a cup," and navigate complex spaces without bumping into walls. For a long time, these robots needed massive amounts of human-written training data to learn how to move, which made them slow to adapt to new houses or offices. Recently, scientists started using giant "brain" models called Large Language Models (LLMs) to give robots common sense, allowing them to figure out navigation on the fly without prior training. However, there's a catch: the smartest brains are locked behind expensive, cloud-based doors that cost money to use and raise privacy worries because they send your home's video feed to a server. While open-source models (free, local brains) exist, they often get confused, forget where they are, or wander in circles, leaving a big gap between what they can do and what the expensive, closed-source ones can achieve.
Enter HiMemVLN, a new approach that teaches open-source robot brains how to remember where they've been and where they're going, effectively curing their "navigation amnesia." The researchers discovered that these open-source robots suffer from two types of forgetfulness: short-term amnesia, where they lose track of the immediate surroundings and end up walking in loops, and long-term amnesia, where they forget the original goal and drift off course after a few steps. To fix this, the team built a "hierarchical memory system" inspired by how human brains work. They created a Short-Term Localer that acts like a visual sketchbook, constantly drawing a map of the rooms it has just visited to spot if it's walking in circles. They also built a Long-Term Globaler that acts like a compass and a mission controller, constantly reminding the robot of the big picture: "You started in the bedroom, you need to go to the kitchen, and you are currently heading the wrong way."
By combining these two memory systems, HiMemVLN allows open-source robots to navigate with a level of reliability that was previously only seen in expensive, closed-source models. In tests, this method didn't just help the robots avoid loops; it nearly doubled their success rate compared to the best previous open-source methods. In simulated environments, the new system achieved a success rate of 30% and a success-weighted path length (SPL) of 26.85, significantly outperforming the previous open-source leader, OpenNav, which only managed 16% success. The researchers also tested this on a real-world wheeled robot in actual rooms, where it again beat the competition, achieving a 32% success rate compared to OpenNav's 18%. The study suggests that by giving robots a structured way to remember both their immediate visual surroundings and their long-term goals, we can build secure, private, and highly capable navigation agents that don't need to rely on the cloud.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.