City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs
This paper introduces **CityNav**, a new benchmark for evaluating the ability of multimodal large language models (MLLMs) to perform knowledge-intensive, real-world city navigation through visual cues alone, and proposes **Verbalization of Path (VoP)** to improve performance by prompting models to explicitly utilize their internal cognitive maps.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you’ve just dropped a very smart, very well-read person into the middle of a massive, unfamiliar city like Tokyo or New York. This person has read every guidebook, every Wikipedia article, and seen millions of photos of these cities online. They are incredibly intelligent, but there’s a catch: they have no GPS, no Google Maps, and no compass. All they have is their eyes and their memory.
This is the problem this research paper is trying to solve.
The Problem: The "Smart but Lost" AI
Current AI models (like the ones behind ChatGPT) are like that person. They are "multimodal," meaning they can "see" images and "read" text. However, when we ask them to actually do something—like navigate a real city—they usually fail.
Most AI testing happens in "simulated" worlds (like a video game) where the rules are simple. But the real world is "wild." It has dead ends, confusing street signs in different languages, and massive distances. Most AI agents get "lost in the sauce"—they forget where they just came from, they get stuck in loops, or they lose track of their goal because they are only looking at the single street corner right in front of them.
The Solution: "Verbalization of Path" (VoP)
The researchers created a new way for AI to navigate called AgentNav, using a clever trick they call Verbalization of Path (VoP).
Think of it this way:
Imagine you are walking through a dark forest. Most AI agents are like people walking with their heads down, only looking at the step immediately in front of them. They might see a tree, but they don't realize they've passed the same mossy rock three times already.
VoP turns the AI into a "Narrator." Instead of just looking at a street corner and picking a direction, the AI is forced to stop and "talk to itself" out loud. It has to say:
- "Okay, I am currently standing at the corner of Broadway and Canal Street."
- "My goal is the One World Trade Center, which is south of here."
- "My plan is to keep walking south on Broadway until I hit Vesey Street, then turn right."
By forcing the AI to verbalize its internal map, the researchers are essentially giving the AI a "mental chalkboard." This prevents the AI from "forgetting" its plan. It’s like the difference between a person wandering aimlessly and a person following a written list of directions they are constantly checking.
The "Memory" Toolkit
To make sure the AI doesn't get dizzy, the researchers gave it three specific types of "mental notes":
- The GPS Log (Decision History): A simple list of everywhere it has turned (e.g., "Left, Left, Right, Straight").
- The "Been Here Before" Alarm (Previous Visit): A way to remember, "Hey, I already tried going left at this intersection and it was a dead end; let's try right this time."
- The Mental Compass (Markovian Memory): A condensed summary of its current situation so it doesn't have to remember every single detail from the last hour, just the important stuff.
The Results: From Lost to Local
The researchers tested this on a massive new dataset called CityNav, covering New York, Tokyo, Vienna, and São Paulo.
They found that even the most advanced AI models (like GPT-4o) are actually quite bad at navigating cities on their own. But when they used the VoP (the "Narrator" method), their success rates skyrocketed. They went from being almost completely lost to successfully navigating complex, multi-kilometer routes through real-world streets.
Summary in a Nutshell
The Paper's Big Idea: Don't just tell an AI to "go there." Tell the AI to "describe where you are, where you're going, and how you plan to get there" every time it hits a crossroads. By turning "thinking" into "talking," we turn a lost wanderer into a capable navigator.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.