🤖 AI

VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memory

The paper introduces VTM-Nav, a training-free framework that leverages a persistent Hierarchical Visual-Topological Memory to effectively reuse cross-episode scene experience for Object-Goal Navigation, significantly outperforming existing memory-reset and text-based baselines without requiring model retraining.

Original authors: Xiaoran Xu, Yupeng Wu, Tianyu Xue, Yifan Xu, Xuanran Dong, Xiaoshan Yang, Changsheng Xu

Published 2026-07-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoran Xu, Yupeng Wu, Tianyu Xue, Yifan Xu, Xuanran Dong, Xiaoshan Yang, Changsheng Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be a helpful housekeeper. In the world of robotics and artificial intelligence, there is a specific challenge called "Object-Goal Navigation." It's like telling a robot, "Go find the red sofa," without giving it a map or a GPS. The robot has to wander around a room, look at what it sees, and use its brain to figure out where things are likely to be.

For a long time, scientists have used "Vision-Language Models" (VLMs) to help these robots. Think of a VLM as a super-smart brain that has read the entire internet and knows that sofas are usually in living rooms and beds are in bedrooms. However, there was a big problem with how these robots were tested. Every time the robot finished a task, scientists would hit a "reset button." The robot would forget everything it just learned about that specific house. If you asked it to find a sofa again in the same house, it would act like it had never been there before, wandering around as if it were the first day of school. This paper asks a simple but powerful question: What if the robot could remember its past adventures in the same house? What if, instead of hitting reset, it could keep a mental map of where it found things before, so the next time it's asked to find a bed, it already knows which room to check first?

The researchers behind this study, known as VTM-Nav, decided to build a robot that doesn't just forget. They created a system that lets a robot keep a "memory bank" of its experiences across different trips through the same house, all without needing to retrain its brain or get help from a human with a map.

Here is how their new system works, using a fun analogy: Imagine the robot is an explorer entering a giant, multi-room mansion. In the old way, every time the explorer left the mansion, they would wipe their memory clean. But VTM-Nav gives the explorer a special, two-part notebook.

The first part of the notebook is a rough sketch of the house's layout. It's like a simple map showing that the "Living Room" connects to the "Hallway," which connects to the "Bedroom." This is the "Topological" part. It doesn't show every chair or picture, just the big rooms and how they are connected.

The second part is a collection of photo albums, one for each room. When the explorer finds a sofa in the Living Room, they don't just remember "sofa." They take a mental snapshot of what the sofa looks like from different angles, note that it was seen through a doorway, and write down, "Hey, I found a sofa here successfully before!" This is the "Visual" part.

When the robot gets a new mission, like "Find a Bed," it doesn't start from scratch. First, it looks at its rough sketch to figure out where it is right now. Then, it flips through its photo albums. It asks, "Which room usually has beds?" The notebook tells it, "The Bedroom!" It then uses that hint to guide its search, looking specifically in the bedroom instead of wasting time in the kitchen. If it sees a bed, it stops. If it gets stuck or can't move forward, a "safety guard" in the system notices and helps it try a different path, but it won't stop the robot if the bed is clearly right in front of it.

The researchers tested this idea in three different virtual worlds (called HM3D v0.1, HM3D v0.2, and MP3D), which are like giant digital houses filled with furniture. They compared their new "memory-keeping" robot against the old "forgetful" robot. The results were pretty clear: the robot with the memory notebook was much better at finding its targets.

On the first test world (HM3D v0.1), the memory robot succeeded 4.6 points more often than the forgetful one. On the second world (HM3D v0.2), it improved by 2.0 points, and on the third world (MP3D), it was 0.8 points better. They also measured how efficiently the robot moved (called SPL), and the memory robot was just as good, or sometimes even better, at not taking unnecessary detours.

Interestingly, they also compared their visual notebook to a robot that just kept a list of text notes (like "I saw a bed in the bedroom"). The visual notebook won again, beating the text-only version by 3.1 points and 5.5 points on the two HM3D tests. This suggests that remembering what things look like and where they are in a room is more useful than just remembering the words.

The paper also looked at what happens as the robot gets more experience. They split the tests into "early" and "late" rounds. The memory robot got 2.7 points better at finding things as it learned more about the house, while the forgetful robot and the text-note robot barely improved at all. This shows that the robot is actually learning from its past trips.

However, the authors are careful to say that this isn't a magic solution for every problem yet. They tested this in controlled simulations with a limit of 40 steps per trip. They didn't claim to have solved navigation forever, but they showed that giving a robot a structured way to remember its past visual experiences in a specific place makes it significantly smarter and more efficient at finding things, without needing to retrain its brain or use a human's map. It's a step toward robots that can truly learn from their daily lives in our homes.

Technical Summary: VTM-Nav

Problem Formulation

The paper addresses Object-Goal Navigation (ObjectNav) in the context of cross-episode experience reuse. While recent training-free agents leverage Vision-Language Models (VLMs) for open-vocabulary navigation, conventional evaluation protocols reset all scene-specific state after every request. In these standard settings, each episode is independently initialized with a random pose and a single target, and any acquired knowledge (connectivity, object viewpoints, successful approaches) is discarded upon termination.

The authors propose a setting where a service robot revisits the same indoor scene for multiple, independently initialized requests. The goal is to determine if an agent with fixed model parameters and navigation components can effectively reuse self-acquired, scene-scoped memory across episodes without retraining or access to oracle information (such as ground-truth maps, shortest paths, or room labels). The challenge lies in re-localizing the agent within a previously acquired scene structure, retrieving target-relevant evidence, and grounding that memory in current observations to guide navigation.

Methodology: VTM-Nav

The authors introduce VTM-Nav, a training-free framework built upon a VLM backbone (specifically Qwen3-VL-Plus) that incorporates a persistent Hierarchical Visual-Topological Memory (VTM). The system operates through a loop of re-localization, memory update, retrieval, and action guidance.

1. Hierarchical Visual-Topological Memory (VTM)

The memory structure is a two-level hierarchy designed to preserve the spatial provenance of evidence:

Coarse Level (Room Topology): A graph $T_s = (V_{room}, E_{room})$ captures observed room connectivity. Each room node stores canonical labels, semantic aliases, visit statistics, and recent visual summaries. This structure supports re-localization and coarse retrieval.
Fine Level (Room-Owned Visual Memory): Each room node owns a local object memory $O_{s,v}$ $O_{s, v}$ , which is further split into:
- In-room evidence ( $O_{in}$ ): Objects directly observed within the room.
- Remote-visible evidence ( $O_{rem}$ ): Objects observed through openings (e.g., doorways), marked with their observer room and potential destination hypotheses.
- Each object node stores category, confidence, observation count, viewpoints, visual summaries, and success-backed approach cues.

2. Online Decision Loop

For each step $t$ in an episode:

Semantic Evidence Generation: The VLM processes the current RGB-D observation to produce a room label, visible object categories, and a textual scene summary.
Room Localization: The agent scores candidate rooms in the stored topology based on semantic (name, object overlap, visual summary), topological (connectivity to the previous room), and spatial (position consistency) consistency. The best-matching room is selected to re-localize the agent.
Memory Update: The agent updates the matched room's visit count and confidence. New rooms or transitions are added to the topology if supported by high-confidence observations. Object nodes are updated or created based on category agreement and visual compatibility.
Hierarchical Retrieval: Given a target category, the system first searches for confirmed in-room evidence in the target-bearing room. If unavailable, it falls back to remote-visible evidence. If a path exists in the topology, a room-hop path (via BFS) is retrieved.
Action Guidance: The retrieved prior (room path, target room, visual cues) is provided to the VLM as soft context. Crucially, the memory guidance can re-rank the navigable candidates derived from the current observation based on lexical agreement with the remembered cues.
Execution Guard: A conservative guard mechanism handles local failures (e.g., blocked motion, oscillations, premature stopping) without overriding successful stops when the target is visible.

3. Episode Termination and Persistence

At the end of an episode, the trajectory and outcome (success/failure) are used to update the persistent scene memory. Successful approaches receive additional confidence increments. The VLM and controller remain fixed; only the memory state persists across episodes.

Key Contributions

Problem Formulation: The paper defines cross-episode ObjectNav as a setting where independently initialized, single-goal episodes share only self-acquired, scene-scoped memory, distinct from continuous goal sequences or standard memory-reset benchmarks.
Hierarchical Visual-Topological Memory: The introduction of a two-level memory structure that couples a coarse room topology with fine-grained, room-owned visual memories. This design preserves the spatial provenance of object evidence and successful approaches, enabling re-localization and target-conditioned retrieval without ground-truth maps.
Training-Free Framework: The proposal of VTM-Nav, which harnesses cross-episode experience through re-localization, coarse-to-fine retrieval, observation-grounded candidate re-ranking, and an execution guard, all without updating the VLM or navigation components.
Empirical Evaluation: A controlled evaluation on HM3D v0.1, HM3D v0.2, and MP3D benchmarks, comparing the hierarchical memory against a memory-reset control and a persistent textual memory baseline.

Experimental Results

The authors evaluated VTM-Nav under a controlled 40-step protocol on three benchmarks, using a matched Qwen3-VL-Plus backbone and navigation components.

Performance Gains:
- HM3D v0.1: VTM-Nav achieved a 59.6% Success Rate (SR), outperforming the memory-reset control (WMNav) by 4.6 points and the textual memory baseline by 3.1 points.
- HM3D v0.2: VTM-Nav reached 72.0% SR, exceeding the memory-reset control by 2.0 points and the textual memory baseline by 5.5 points.
- MP3D: VTM-Nav achieved 44.3% SR, surpassing the memory-reset control (43.5%) and textual memory (43.3%).
- SPL (Success weighted by Path Length): VTM-Nav maintained comparable or higher SPL across all benchmarks, indicating efficient path planning.
Ablation Studies:
- Removing the room topology reduced SR to 56.5 (on HM3D v0.1), highlighting the importance of structural indexing.
- Removing persistence (clearing memory between episodes) resulted in 59.4% SR, isolating the value of cross-episode reuse versus within-episode working memory.
- Removing the execution guard or visual object evidence led to performance drops, confirming their roles in handling local failures and providing fine-grained cues.
Cross-Episode Progression: Analysis of early vs. late episodes in HM3D v0.2 showed that VTM-Nav improved its SR by 2.7 points as scene experience accumulated, whereas the memory-reset and textual memory baselines showed negligible improvement (0.7 and 0.3 points, respectively).

Significance and Claims

The paper claims that hierarchical visual-topological memory is an effective representation for reusing cross-episode scene experience in ObjectNav. The results demonstrate that an agent can significantly improve navigation success by retaining and structuring scene knowledge (room connectivity, object viewpoints, and successful approaches) without retraining the underlying VLM or requiring oracle information.

The authors emphasize that their approach supports re-localization and target-conditioned retrieval in a setting where physical state continuity is broken between episodes. The comparison with persistent textual memory suggests that structured visual-topological organization is superior to free-form text summaries for guiding navigation in recurring environments. The work validates the feasibility of training-free, experience-reusing agents for service robotics in repeated indoor deployments.