MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

Imagine you are trying to build a robot that can act as a super-smart travel guide. You want it to tell you how far it is from your house to the nearest pizza shop, or which direction you need to walk to find a specific park. This is the world of geospatial reasoning: the ability to understand where things are, how far apart they are, and how they fit together on a map.

For a long time, scientists have been teaching computers to "read" maps and facts about the world. They've built massive digital libraries of information called Knowledge Graphs, which are like giant, interconnected webs of facts (e.g., "Paris is in France," "The Eiffel Tower is in Paris"). Recently, we've also created Large Language Models (LLMs)—super-powered AI brains that can read almost everything written on the internet and answer questions in human language. The big question everyone is asking is: If you give these AI brains a map and a list of facts, can they actually do the math to figure out distances and directions, or do they just guess based on what they've memorized?

This is where a new study called MultiGlobeQA steps in. The researchers wanted to see if these AI brains are truly smart enough to navigate the real world or if they are just "hallucinating" (making things up) when asked to do geometry. They built a giant, multilingual test to see exactly where these models fail.

The Great Map Test: MultiGlobeQA

The researchers created a massive challenge called MultiGlobeQA. Think of it as a giant, global "test drive" for AI, but instead of driving a car, the AI has to solve 46,060 different questions about real places on Earth.

To make sure the test was fair and covered the whole world, they didn't just pick famous cities like London or New York. They used a special "stratified sampling" method, which is like taking a giant scoop of ice cream that includes every flavor and every layer, ensuring they got questions from 201 different countries and territories. They made sure to include places that are rich and poor, crowded and empty, so the test wouldn't just favor the places where the internet is best.

They also made the test multilingual. The same questions were translated into 17 different languages, ranging from English and Spanish to Urdu and Georgian. This was crucial because they wanted to see if the AI got confused just because the question was asked in a different language, or if the problem was the math itself.

The questions covered 14 different types of "spatial thinking," such as:

• Distance: "How far is it from Town A to Town B?"
• Direction: "If I walk from the airport to the museum, am I going North or South?"
• Shape: "Is this country shaped like a rectangle or a circle?"
• Grid Indexing: "What is the specific code (like a digital address) for this spot on the map?"

Crucially, every single answer in this test wasn't just guessed by a human. It was computed by a computer program using real math and geometry. This means the "correct" answer was verified by the laws of physics and math, not by opinion.

The Results: Smart at Facts, Clueless at Math

When the researchers ran their AI models through this test, the results were a bit of a shock.

1. The "I Know It, But I Can't Do It" Problem
The AI models were surprisingly good at remembering facts. If you asked, "What is the capital of France?", they got it right almost every time. They even had the coordinates (the exact numbers for where a place is) stored in their memory.

However, when it came to doing the math with those numbers, they collapsed.

• When asked to calculate the distance between two towns, the AI often guessed wildly wrong. In one example, the real distance was about 19 kilometers, but the AI guessed 1.4 kilometers. That's like guessing a 20-minute walk is only a 2-minute stroll!
• When asked to figure out a "geohash" (a specific code that pinpoints a location on a grid), the models failed miserably, even when the researchers gave them the exact facts they needed.

2. The "Tool Use" Trap
The researchers tried giving the AI a "calculator" (a tool to do the math) and letting it search the web or a database for the facts. This helped a lot, but not as much as you'd hope.

• Even when the AI had the perfect facts handed to it on a silver platter (called "gold triples") and a perfect calculator, it still couldn't solve about one-third of the questions.
• The models were great at simple things like "Is City A inside Country B?" (topology) or "Which way is North?" (direction).
• But they completely fell apart on things that required grid indexing (pinning a location to a digital grid) or shape computation. On these hard tasks, their success rate stayed below 30%, even with all the help.

3. The Rich vs. Poor Gap
The study also found a sad inequality. The AI models were much better at answering questions about high-income countries (wealthier nations) than low-income countries.

• This wasn't because the AI was "racist" or "biased" in a human way, but because the digital maps and data they were trained on are much fuller and more detailed for rich countries.
• Even when the researchers gave the AI the perfect facts for a poor country, it still struggled more than it did for a rich country. This suggests the problem isn't just a lack of data; the AI just isn't good at doing the math for places it hasn't "seen" as much before.

4. Language Didn't Matter Much
Interestingly, the language didn't make a huge difference. Whether the question was in English, Russian, or Vietnamese, the AI's performance was roughly the same. The problem wasn't that the AI couldn't understand the words; it was that it couldn't do the geometry.

The Big Takeaway

The main lesson from MultiGlobeQA is that having a lot of knowledge isn't the same as being able to reason.

Imagine you have a student who has memorized every street name in the world and every distance between cities. If you ask them, "How far is it from A to B?", they might just guess a number that sounds right because they've heard those names before. But if you ask them to actually calculate the distance using a ruler and a map, they might fail because they don't know how to use the ruler.

The paper suggests that for AI to be truly useful for things like navigation, logistics, or disaster response, we can't just feed it more facts. We need to teach it how to compute and reason over those facts. Currently, the bottleneck isn't that the AI doesn't know the world; it's that the AI is terrible at doing the math to understand the world.

Even with the best tools and the most perfect data, the current AI models are still stumbling over basic spatial puzzles. It's a reminder that while these models are amazing at talking and remembering, they are still learning how to truly "think" about space.