TrajRAG: Retrieving Geometric-Semantic Experience for Zero-Shot Object Navigation

Technical Summary: TrajRAG – Retrieving Geometric-Semantic Experience for Zero-Shot Object Navigation

Problem Statement
Existing zero-shot Object Goal Navigation (ObjectNav) methods often rely on commonsense knowledge derived from large language models (LLMs) or vision-language models (VLMs). However, this knowledge is typically grounded in internet-scale text rather than embodied 3D spatial experience. Furthermore, while some methods utilize episodic observations collected during navigation, these are often discarded after each episode, preventing the accumulation of lifelong experience. Consequently, agents struggle with redundant exploration and lack the ability to retrieve scene-specific geometric-semantic priors to guide decision-making in unseen environments.

Methodology
The authors propose TrajRAG (Trajectory RAG), a retrieval-augmented generation framework designed to build a "long-term" memory of navigation experiences. The system operates through three core mechanisms:

1. Topological-Polar (Topo-Polar) Trajectory Representation:
   To address the redundancy in raw RGB-D observation trajectories (caused by revisits, loops, and spatial overlaps), TrajRAG converts continuous observations into a compact, structured representation.

   • Skeletonization: A semantic map is constructed from RGB-D data, and the navigable region is skeletonized to extract topological nodes.
   • Polar Encoding: For each topological node, the surrounding space is discretized into 12 polar sectors (30° each). Each sector records the observed semantic labels (object, obstacle, unknown, or free). This creates a "fingerprint" of the local geometric-semantic layout that is robust to agent heading differences.
   • Pruning: The system merges consecutive identical nodes and prunes temporal loops to ensure stored trajectories are efficient and loop-free.

2. Hierarchical Chunking and Retrieval Architecture:
   TrajRAG organizes these topo-polar trajectories into a hierarchical structure to enable efficient coarse-to-fine retrieval:

   • Chunks: Each chunk corresponds to a single topo-polar trajectory, including its topological structure, a natural language description, and a learned embedding.
   • Coarse Index (Summaries): Trajectories with similar topological structures are grouped and merged into unified "topo-polar summaries." These summaries act as descriptors for the group's overall layout.
   • Fine Index (Trajectory Encoder): A specialized trajectory encoder (combining an encoder-only transformer for node embeddings and a decoder-only transformer for sequential dependencies) embeds individual trajectories. This encoder is trained via contrastive learning to ensure trajectories with the same navigation goal lie close in the embedding space.
   • Retrieval Process: During navigation, candidate exploration paths generate trajectory hypotheses. The system first performs coarse matching against topo-polar summaries to identify relevant groups, followed by fine-grained matching within those groups to retrieve specific trajectory chunks.

3. Navigation and Incremental Learning:

   • Inference: At each timestep, the agent constructs a semantic map and identifies frontiers. Candidate paths are generated, and TrajRAG retrieves relevant past experiences (geometric-semantic layouts) to assist the planner (e.g., an LLM) in selecting the optimal next waypoint.
   • Accumulation: After an episode concludes, the new trajectory is incrementally integrated into TrajRAG. It undergoes semantic and geometric matching against existing summaries. If it matches an existing group, it updates the summary; if not, a new group is created. This allows the system to accumulate lifelong navigation experience.

Key Contributions

• Topo-Polar Representation: A novel method to compactly encode spatial layouts and semantic contexts, effectively removing redundancies in raw episodic observations while preserving distinctive geometric-semantic fingerprints.
• Hierarchical Retrieval Framework: A coarse-to-fine retrieval strategy that balances efficiency and accuracy by first filtering by scene layout (summaries) and then by specific trajectory details (embeddings).
• Lifelong Experience Accumulation: A mechanism to continuously consolidate episodic memories into a long-term knowledge base, enabling the agent to learn from past interactions across multiple episodes and scenes.
• Bridging Common Sense and Experience: The framework integrates scene-agnostic reasoning from large models with scene-specific geometric-semantic experience retrieved from the agent's own history.

Experimental Results
The authors evaluated TrajRAG on zero-shot ObjectNav benchmarks using the MP3D, HM3D-v1, and HM3D-v2 datasets within the Habitat simulator.

• Performance: TrajRAG achieved state-of-the-art performance across all benchmarks, outperforming existing LLM/VLM-based methods and other RAG formulations (such as GraphRAG and text-based RAG).
  • On HM3D-v2, it achieved a Success Rate (SR) of 78.1% and Success weighted by Path Length (SPL) of 40.2%.
  • On HM3D-v1, it achieved an SR of 62.5% and SPL of 33.9%.
  • On MP3D, it achieved an SR of 42.6% and SPL of 18.0%.
• Ablation Studies: Experiments confirmed that the combination of geometric and semantic cues in the topo-polar representation is superior to using text-only or geometry-only features. Additionally, the hierarchical retrieval strategy (coarse-to-fine) was shown to be critical, as removing the coarse stage or using generic text embeddings significantly degraded performance.
• Cross-Dataset Generalization: The retrieval corpus built from one dataset (e.g., HM3D-v1) effectively guided navigation in another (e.g., MP3D), demonstrating the transferability of the learned topological-semantic representations.

Significance
The paper claims that TrajRAG addresses a fundamental limitation in current zero-shot ObjectNav: the lack of accumulated, scene-specific experience. By retrieving geometric-semantic experiences rather than relying solely on pre-trained commonsense, TrajRAG enables agents to make more informed decisions in complex, unseen environments. The work demonstrates that bridging the gap between large-model reasoning and embodied 3D experience through a structured, lifelong memory system significantly enhances navigation accuracy and efficiency. The authors position this as a step toward continual learning in embodied agents, where short-term memories are gradually consolidated into long-term knowledge to support future decision-making.