💻 computer science

LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

This paper introduces LangMap, the first real-world 3D indoor navigation benchmark featuring human-verified hierarchical semantic annotations across four goal levels, alongside the PlaNaVid baseline, to address limitations in existing open-vocabulary language-conditioned goal navigation evaluations.

Original authors: Bo Miao, Weijia Liu, Jun Luo, Lachlan Shinnick, Jian Liu, Thomas Hamilton-Smith, Yuhe Yang, Zijie Wu, Vanja Videnovic, Feras Dayoub, Anton van den Hengel

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Miao, Weijia Liu, Jun Luo, Lachlan Shinnick, Jian Liu, Thomas Hamilton-Smith, Yuhe Yang, Zijie Wu, Vanja Videnovic, Feras Dayoub, Anton van den Hengel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-stakes game of "Hide and Seek" inside a giant, complex house, but you are blindfolded and can only see what's directly in front of you. A friend is shouting instructions over a walkie-talkie, telling you where to go.

This paper introduces a new, much harder, and much fairer version of that game, along with a new strategy for playing it.

The Problem: The "Bad Map" Problem

In the past, researchers tried to teach robots to find things based on language. But they had a major flaw: the "maps" (the instructions) they used were often written by AI (Vision-Language Models) that didn't quite understand the house.

Think of it like this: If you asked an AI to describe a specific red chair in a room full of chairs, it might say, "Find the red chair." But if there are three red chairs, that instruction is useless. The AI might also get confused and say, "Find the chair near the window," when the window is actually in a different room. The paper found that nearly 40% of these AI-generated instructions were wrong or confusing. It's like trying to navigate a city using a map drawn by someone who has never left their bedroom.

The Solution: LangMap (The Human-Verified Map)

To fix this, the authors created LangMap. Instead of letting an AI guess the instructions, they hired humans to look at real 3D scans of houses and write the instructions.

They didn't just stop at "Find a chair." They created a four-level hierarchy of difficulty, like a video game with increasing levels:

Scene Level (The Easy Mode): "Find any chair in the whole house." (Like looking for a needle in a haystack, but you don't care which needle).
Room Level (Medium Mode): "Find a chair in the kitchen." (Now you have to know what a kitchen looks like).
Region Level (Hard Mode): "Find a chair in the bedroom with the blue rug." (Now you have to distinguish between two different bedrooms).
Instance Level (Expert Mode): "Find the specific chair with the scratch on the left leg, next to the window." (This requires spotting tiny details and knowing exactly which object is which).

The authors call this HieraNav (Hierarchical Navigation). They built a massive dataset with over 18,000 tasks covering 414 different types of objects. Every single instruction was checked by a human to make sure it was unique and accurate. It's the difference between a blurry photocopy of a map and a high-definition, hand-drawn treasure map.

The New Player: PlaNaVid

The paper also introduces a new robot player called PlaNaVid.

Most robots trying to solve this problem rely on expensive 3D sensors (like LiDAR) or deep depth cameras to build a 3D model of the room. It's like the robot is wearing a high-tech helmet that lets it see the "skeleton" of the room.

PlaNaVid is different. It only has a standard camera (RGB), just like a human eye. It doesn't build a 3D model, and it doesn't know the exact distance to objects. So, how does it win?

It uses a clever trick called Bounded Diverse Memory (BDM).

The Analogy: Imagine you are walking through a maze. Instead of trying to remember every single step you took (which is too much for your brain), you take a few snapshots of the most interesting or different parts of your journey.
The Strategy: When the robot needs to decide where to go next, it doesn't just look at what's in front of it. It "flips through" its mental photo album of the last few unique places it visited. It uses a smart planner (an AI brain) to look at these photos and say, "Ah, I remember seeing a hallway that leads to the kitchen in this photo. Let's go that way."

This allows the robot to plan ahead and navigate complex, multi-step tasks (like "Find a cup, then go to the coffee machine, then find a book") without needing expensive 3D hardware.

The Results

When they tested this new setup:

The Map: The human-written instructions were far superior to the AI-written ones. In a test to see if a computer could match a description to the right object, the human-written instructions were correct 81% of the time, while the old AI instructions were only correct 58% of the time.
The Robot: PlaNaVid, using only a standard camera and its "photo album" memory, beat almost every other robot on the market. It succeeded in finding targets 42.6% of the time in complex multi-step tasks, outperforming robots that used expensive 3D sensors.

What's Still Hard?

Even with this new map and robot, the paper admits there are still tough challenges:

The "Long Tail": If the robot is asked to find a very rare object (like a specific type of vintage toaster), it struggles because it hasn't seen many examples.
Small and Distant Objects: If the target is tiny or far away, it's hard to spot.
The Chain Reaction: If a robot fails the first step of a multi-step task (e.g., it can't find the cup), the whole mission fails. This is still very difficult to solve.

Summary

In short, the authors built a better, human-verified map for robot navigation that tests robots on everything from "find a chair" to "find that specific scratched chair in the blue room." They also built a smart robot that can navigate this complex world using only a regular camera and a clever memory system, proving you don't need expensive 3D sensors to be a good navigator.

Technical Summary: LangMap and HieraNav

Problem Definition

Language-conditioned goal navigation (LGN) requires embodied agents to locate user-specified targets in 3D environments without step-by-step guidance. While existing benchmarks focus primarily on category-level goals (e.g., "find a chair") or rely on instance descriptions generated by Vision-Language Models (VLMs), these approaches suffer from significant limitations:

Semantic Ambiguity: VLM-generated descriptions often lack uniqueness, containing semantic errors or failing to distinguish between visually similar instances within the same scene.
Limited Granularity: Current benchmarks rarely support goals specified at finer semantic levels (region or instance) or mixed-level multi-goal episodes.
Evaluation Reliability: The noise in automated annotations hinders systematic and reliable evaluation of agent performance, particularly for tasks requiring fine-grained spatial and semantic reasoning.

Methodology

1. LangMap Benchmark and HieraNav Task

The authors introduce LangMap, a real-world 3D indoor navigation benchmark built upon HM3D scans, and HieraNav, a hierarchical open-vocabulary goal navigation task.

Four Semantic Levels: HieraNav defines goals across four distinct levels of granularity:
- Scene-level: Any object of a target category in the scene.
- Room-level: An object of a target category within a specific room type.
- Region-level: An object in a specific room instance, distinguished by contextual cues (e.g., "bedroom with a geometric rug").
- Instance-level: A unique object instance identified by discriminative attributes or relations (e.g., "the phone on the Bluey bed").
Contrastive Annotation Protocol: To ensure high-quality, human-verified data, the authors employ a rigorous contrastive annotation process:
- Annotators compare same-category regions and instances within a scene to compose discriminative descriptions that uniquely identify targets.
- Descriptions are categorized into concise (emphasizing salient cues) and detailed (providing richer context) styles.
- The dataset covers 414 object categories, with 349 used to construct over 18,000 navigation tasks.
Statistics: LangMap includes 926 discriminative region descriptions and 7,510 instance descriptions. It features a higher proportion of small objects (22.2% with average IoU < 3.3%) compared to prior benchmarks, offering a more realistic evaluation of localization capabilities.

2. PlaNaVid Baseline

To address the lack of strong baselines for multi-goal navigation without depth sensors or 3D scene representations, the authors propose PlaNaVid, a decoupled, RGB-only framework.

Architecture: The system consists of two stages:
1. Memory-Guided Planning: An LLM planner (Qwen2.5-VL-7B) selects an initial waypoint and heading for the current goal. It is primed by a Bounded Diverse Memory (BDM) module.
2. Reactive Policy: A reactive navigation policy (Uni-NaVid) executes low-level actions based on RGB observations.
Bounded Diverse Memory (BDM): To handle long-horizon episodes (up to 2,500 steps) within limited LLM context windows, BDM maintains a bounded set of RGB snapshots ( $N_{max}=50$ $N_{ma x} = 50$ ).
- Global-Uniform Update (GUU): Ensures near-uniform temporal coverage by pruning entries with the smallest local temporal gaps when the buffer is full.
- Semantic-Diverse Retrieval (SDR): Uses a greedy max-min strategy to select a subset of memory states that maximizes semantic diversity, providing the planner with informative context without redundancy.
Constraints: PlaNaVid operates without depth inputs, 3D scene representations (e.g., occupancy maps), or object masks.

Key Results

Benchmark Evaluation

Annotation Quality: In text-to-view matching tasks, LangMap's human-verified instance descriptions achieved 81.3% accuracy (using ~5 words) compared to 58.4% for GOAT-Bench's VLM-generated descriptions (using ~21 words). LangMap descriptions were non-inferior in 94% of cases and significantly more concise.
Navigation Performance:
- PlaNaVid achieved the highest Success Rate (SR) among evaluated methods: 42.6% for multi-goal navigation and 31.4% for single-goal navigation.
- It outperformed the previous best RGB-only method (Uni-NaVid) by 8.5% in multi-goal SR and 4.0% in Sequence Success Rate at 2 (SeqSR@2).
- It surpassed 3D-based methods (e.g., MTU3D) in multi-goal SR despite lacking depth inputs, validating the efficacy of the memory-guided planning approach.

Ablation Studies

Memory Components: Removing BDM reduced multi-goal SR from 42.6% to 34.1%. The combination of GUU and SDR was critical, compressing memory by 13× (from ~656 frames to 50) without performance loss.
Description Styles: Detailed descriptions improved performance at region and instance levels, though concise descriptions remained effective.
Challenges: Performance consistently degraded for long-tail categories, small objects (low visibility), distant targets, and multi-goal completion sequences, highlighting these as open challenges.

Significance and Claims

The paper claims that LangMap and HieraNav establish a rigorous, human-verified testbed for language-driven embodied navigation, addressing the critical gap of noisy automated annotations and limited semantic granularity in prior work. By providing discriminative descriptions across four hierarchical levels, the benchmark enables systematic evaluation of agents' ability to perform spatiotemporal reasoning and disambiguation.

Furthermore, PlaNaVid demonstrates that high-performance multi-goal navigation is achievable using only RGB inputs and memory-guided planning, challenging the necessity of depth sensors and explicit 3D scene reconstruction for complex navigation tasks. The authors position their work as a foundation for future research into long-horizon reasoning, handling of long-tail categories, and robust multi-goal completion in real-world environments.