GeoNav: Empowering MLLMs with dual-scale geospatial reasoning for language-goal aerial navigation

This paper introduces GeoNav, a multi-modal agent that enhances MLLMs for long-range aerial navigation by employing a three-phase, coarse-to-fine reasoning process supported by dual-scale spatial representations (a global cognitive map and a local scene graph) and a spatial chain-of-thought mechanism, achieving state-of-the-art performance on the CityNav benchmark.

The Gist

Imagine you are asking a friend to find a specific house in a massive, unfamiliar city using only a voice message. You say: "Find the gray house with a red car parked in front of it, located on Wellington Road, near the train station."

If you gave this task to a standard robot drone, it would likely get confused. It might fly in circles, stare at every gray house, or get lost because the city is too big to see all at once.

GeoNav is a new "super-smart" drone pilot designed to solve exactly this problem. It doesn't just look at pictures; it thinks like a human explorer. Here is how it works, broken down into simple concepts:

1. The Two-Brain System (Dual-Scale Reasoning)

Most drones try to remember every single brick and leaf they see, which is overwhelming. GeoNav uses a clever two-part memory system, like a human using both a map and a mental sketch.

• The "Schematic Map" (The Big Picture):
  Think of this as a simplified, hand-drawn map you might see in a travel brochure. It doesn't show every tree, but it shows the big landmarks: "The train station is here, the park is there."

  • What it does: This helps the drone fly quickly across the city to the general neighborhood (e.g., "Go to the train station"). It ignores the tiny details to focus on the big direction.

• The "Scene Graph" (The Detailed Sketch):
  Once the drone arrives near the train station, it switches to a different mode. Imagine a family tree or a flowchart that connects things: "The library is next to the park. The red car is parked behind the library."

  • What it does: This is a detailed web of relationships. It helps the drone distinguish between that gray house and the other gray house by checking who is standing next to whom.

2. The Three-Step Detective Process

Instead of trying to find the target in one giant leap, GeoNav breaks the job into three distinct phases, mimicking how a human detective solves a case:

• Phase 1: The Commute (Landmark Navigation)

  • Analogy: You tell a taxi driver, "Take me to the downtown library." You don't ask the driver to find a specific window on the 4th floor yet.
  • Action: The drone uses its "Schematic Map" to fly straight to the general area (the landmark) as fast as possible.

• Phase 2: The Scouting (Target Search)

  • Analogy: Now that you are at the library, you start walking around, looking at the buildings nearby. You are curious and scanning the area.
  • Action: The drone hovers near the landmark, looking for objects that match your description (like a red car). It starts building its "Scene Graph" (the detailed sketch) of what it sees.

• Phase 3: The Pinpoint (Precise Localization)

  • Analogy: You spot a gray house, but wait—is it the right one? You check your notes: "Is there a red car in front?" "Yes! That's the one."
  • Action: The drone uses its detailed "Scene Graph" to query the relationships. It asks, "Which gray house has a red car next to it?" Once it finds the match, it lands.

3. The "Chain of Thought" (Talking to Itself)

The drone uses a powerful AI brain (a Multi-Modal Large Language Model) that doesn't just guess. It talks to itself before moving.

• How it works: Before it moves, the AI says: "Okay, I am 100 meters away from the library. The instruction says the target is to the North. Therefore, I should move North."
• Why it matters: This "reasoning" step prevents the drone from making random mistakes. If it gets stuck, it can say, "Hmm, I can't find the red car. Maybe I should look at the building to the left instead."

4. Why This is a Big Deal

Previous drones were like a person trying to find a needle in a haystack by looking at one straw at a time. They often got lost in big cities.

GeoNav is like a person who:

1. Looks at a map to get to the right neighborhood.
2. Walks around the neighborhood looking for clues.
3. Uses logic to figure out exactly which house is the right one.

The Result: In tests, this method was 18% more successful than the best existing methods. It didn't just find the target; it found it faster and with fewer wrong turns.

Summary

GeoNav is a drone that stops trying to "see everything at once" and starts thinking in stages. It uses a big map to get to the neighborhood, a detailed sketch to find the specific object, and a logical inner voice to make sure it's making the right move. It turns a confusing, chaotic search into a structured, step-by-step adventure.

Technical Summary

Here is a detailed technical summary of the paper "GeoNav: Empowering MLLMs with Dual-Scale Geospatial Reasoning for Language-Goal Aerial Navigation."

1. Problem Definition

The paper addresses Language-Goal Aerial Navigation, a task where an Unmanned Aerial Vehicle (UAV) must navigate to a specific target in complex urban environments based on natural language instructions (e.g., "Find a house with a gray roof on Wellington Road with a red car parking in front of it").

Key Challenges:

• Ambiguous Visual Semantics: Urban environments contain structurally similar objects (e.g., many identical buildings or cars), making identification difficult without geospatial context.
• Multi-scale Spatial Reasoning: The task requires reasoning across macro-scale elements (road networks, districts) and micro-scale entities (specific vehicles, building features). Existing methods often fail to integrate these levels.
• Long-horizon Planning: Targets are often out of view initially, requiring a multi-phase approach (navigation $\to$ search $\to$ localization) rather than end-to-end direct navigation.
• Limitations of Current Approaches: Indoor navigation methods (egocentric vision-language matching) do not scale to urban settings. Existing aerial methods often lack effective spatial reasoning capabilities or fail to utilize geographic priors (landmarks).

2. Methodology: GeoNav Framework

GeoNav is a zero-shot agentic framework that bridges urban geographic knowledge and embodied perception using Multi-Modal Large Language Models (MLLMs). It operates on a coarse-to-fine workflow inspired by human spatial reasoning, utilizing a Dual-Scale Spatial Memory system.

A. Three-Stage Navigation Strategy (MNS)

The task is decomposed into three sequential stages managed by a Stage Scheduler:

1. Landmark Navigation: The UAV navigates toward a coarse-grained geographic landmark (e.g., "near the train station") using global priors.
2. Target Search: Once near the landmark, the UAV performs curiosity-driven local exploration to find objects matching the description.
3. Precise Localization: The system pinpoints the exact target location and executes a stop command.

B. Dual-Scale Spatial Memory

To support reasoning across scales, GeoNav maintains two distinct memory structures:

1. Schematic Cognitive Map (SCM):
   • Scope: Global but schematic.
   • Function: Fuses prior geographic knowledge (landmark contours) with real-time top-down visual observations.
   • Representation: A unified 2D visual map annotated with landmarks, the UAV's trajectory, and observed objects. It provides intuitive direction and distance cues for long-range navigation.
2. Hierarchical Scene Graph (HSG):
   • Scope: Local but delicate.
   • Function: Constructed dynamically during the "Search" stage around the landmark.
   • Structure: Contains three node types: Blocks (large areas), Landmarks (POIs), and Objects (detected entities).
   • Edges: Represent spatial relationships (e.g., "contains," "north of," "next to") derived from rule-based functions and MLLM reasoning.
   • Usage: Used for precise target retrieval via multi-hop queries.

C. Geospatial Reasoning & Action Generation

• Stage-Conditioned Reasoning: The MLLM is prompted with stage-specific rationales (e.g., distance to landmark for navigation; visual field tips for search).
• Chain-of-Thought (CoT): The MLLM generates a textual reasoning process ($\Gamma_t$) before outputting an action ($A_t$), ensuring interpretability.
• Target Retrieval: In the localization stage, the system converts natural language instructions into an Operation Chain (e.g., get_geonode $\to$ filter_edge $\to$ filter_by_class). If the initial query fails, a Fallback Mechanism relaxes constraints (e.g., changing "adjacent to" to "contains") to ensure the target is found.

3. Key Contributions

1. Stage-Aware Scheduling: A novel mechanism that decomposes long-horizon aerial navigation into manageable sub-goals (Navigate, Search, Localize) using structured multi-modal rationales.
2. Dual-Scale Spatial Representations: The integration of a global Schematic Cognitive Map (SCM) for fast navigation and a local Hierarchical Scene Graph (HSG) for accurate localization, effectively fusing geographic priors with visual observations.
3. LLM-Driven Target Retrieval: A robust query mechanism with a fallback strategy that allows the agent to handle complex spatial descriptions and recover from query failures.
4. State-of-the-Art Performance: Demonstrated significant improvements on the CityNav benchmark, a challenging urban navigation dataset.

4. Experimental Results

Experiments were conducted on the CityNav dataset (Birmingham and Cambridge urban blocks), comparing GeoNav against rule-based methods, learning-based baselines (Seq2Seq, CMA, MGP), and zero-shot MLLMs (NavGPT, Qwen-VL).

• Success Rate (SR): GeoNav achieved a 25.9% success rate on the "Test Unseen" split, surpassing the second-best method (TopV-Nav) by 18.4%.
• Navigation Error (NE): Reduced the average error to 73.5m, significantly better than most baselines.
• Success weighted by Path Length (SPL): Achieved 16.0%, leading by 10.7% over the second-best.
• Generalization: GeoNav showed superior performance across Easy, Medium, and Hard difficulty levels, particularly excelling in "Hard" tasks where other methods failed (SR 16.7% vs. 2.1% for the next best).
• Ablation Studies: Confirmed that removing either the SCM, HSG, or the Stage Scheduler (MNS) caused significant performance drops, validating the necessity of the dual-memory and staged approach.
• Efficiency: By invoking the MLLM only once every 10 steps (rather than per action), the system maintains real-time feasibility while reducing token consumption.

5. Significance

• Bridging the Gap: GeoNav effectively addresses the gap between indoor navigation methods and the complex requirements of urban aerial navigation, proving that geographic priors are essential for large-scale tasks.
• Interpretability: The use of structured memory (SCM/HSG) and Chain-of-Thought reasoning makes the decision-making process transparent and debuggable, unlike black-box end-to-end models.
• Practical Deployment: The framework demonstrates that MLLMs can be deployed for real-world UAV tasks by decoupling high-level reasoning from low-level control, balancing performance with computational efficiency.
• Future Direction: The work highlights the potential of "coarse-to-fine" reasoning in embodied AI and sets a new baseline for language-goal aerial navigation, pointing toward future research in adaptive altitude control and end-to-end spatial memory unification.