ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps

This paper introduces ReasonMap, a new benchmark comprising high-resolution transit maps and diverse question-answer pairs to evaluate multimodal large language models' fine-grained visual reasoning capabilities, revealing that open-source base models often outperform their reasoning-tuned variants while emphasizing the critical need for direct visual grounding over language priors.

The Gist

Imagine you are trying to teach a robot how to navigate a city using only a giant, high-definition subway map. You point to two stations and ask, "How do I get from here to there?"

For a human, this is easy. You trace the colored lines with your finger, count the stops, and maybe figure out where you need to switch trains. But for an Artificial Intelligence (AI), this is surprisingly difficult. It's like asking someone who has read a million books about London to navigate the London Underground without ever having seen a map. They might know the names of the stations, but they can't "see" the connections.

This paper introduces REASONMAP, a new "driving test" for AI models designed specifically to see if they can actually look at a map and reason through a journey, rather than just guessing based on what they've memorized.

Here is the breakdown of the paper using some everyday analogies:

1. The Problem: The "Bookworm" vs. The "Navigator"

Current AI models are like bookworms. They have read so much text that they know facts about the world. If you ask, "What is the capital of France?", they answer instantly. But if you show them a complex subway map and ask for a route, they often fail.

Why? Because they tend to rely on their "inner knowledge" (what they remember from training) instead of actually looking at the image in front of them. It's like a student who memorized the answers to a practice test but fails the real exam because the questions are slightly different.

2. The Solution: The "Subway Exam" (REASONMAP)

The researchers built a massive dataset called REASONMAP. Think of this as a giant, 30-city subway exam.

• The Map: They collected high-resolution maps from 30 cities (like New York, London, Singapore, and Beijing). These aren't blurry sketches; they are crisp, detailed images full of tiny text and colorful lines.
• The Questions: They created over 1,000 questions. Some are simple: "How do I get from A to B?" Others are harder: "How many stops are in between?" or "List every station I pass."
• The Difficulty: Just like a real exam, some questions are easy (direct line), some are medium (one transfer), and some are hard (multiple transfers on a crowded map).

3. The Surprise Discovery: The "Overthinker" Paradox

The researchers tested 16 different AI models, including both "open-source" (free for anyone to use) and "closed-source" (proprietary, like the big tech company models).

They found a weird, counter-intuitive pattern:

• Open-Source Models: The "smart" versions of these models (the ones trained specifically to "think" step-by-step) actually performed worse than their simpler "base" versions.
  • The Analogy: Imagine a student who is told to "think hard" about a math problem. Instead of solving it, they start doubting themselves, changing their answer five times, and eventually getting it wrong because they over-analyzed the map. They got confused by their own thoughts.
• Closed-Source Models: The opposite happened. The "smart" versions of the big tech models performed better.
  • The Analogy: These models are like a seasoned navigator. Even if they get confused for a second, they can look back at the map, correct their mistake, and find the right path. They have better "visual grounding"—they actually trust what they see.

4. The "Blindfold" Test

To prove that AI needs to see the map, the researchers did a "blindfold test." They gave the AI the question but hid the map, letting it rely only on its memory.

• Result: Most models crashed. Their performance dropped significantly.
• The Lesson: This proves that for tasks like reading a map, you can't just rely on what you've memorized. You need to actually look at the visual details. If the AI can't "see" the lines, it can't solve the puzzle.

5. The Training: Teaching the AI to "Think" Better

Finally, the researchers tried to fix the problem. They used a technique called Reinforcement Fine-Tuning.

• The Analogy: Imagine a coach giving a player feedback after every play. "Good job finding the station, but you missed the transfer point. Next time, look closer."
• By rewarding the AI for getting the route right and punishing it for formatting errors or hallucinations, they trained the models to be more accurate. The AI got better at navigating the "subway" without needing to be a genius, just a careful observer.

Why Does This Matter?

This isn't just about subways. If an AI can't read a subway map, it can't:

• Help a blind person navigate a city.
• Plan routes for self-driving cars in complex traffic.
• Understand medical diagrams or engineering blueprints.

REASONMAP is a wake-up call. It tells us that making AI "smarter" with more data isn't enough. We need to teach them how to look at the world and connect the dots visually, not just verbally. It's the difference between a robot that talks about a map and a robot that can actually use one.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) have shown significant progress in semantic understanding and text-image alignment. However, their ability to perform fine-grained visual reasoning combined with spatial planning remains underexplored. Existing benchmarks often focus on:

• Symbolic/Mathematical reasoning (e.g., MathVQA, MMMU) where visual understanding is limited.
• Coarse-grained spatial tasks (e.g., CityBench, MapBench) that lack the detail required for precise navigation.
• Abstract visual puzzles that do not reflect real-world complexity.

There is a critical gap in evaluating how well MLLMs can interpret high-resolution, information-dense visual artifacts (like transit maps) to perform multi-step spatial reasoning (e.g., finding a route, identifying transfer points, and counting stops) without relying on external tools or prior knowledge.

2. Methodology

A. Dataset Construction (REASONMAP)

The authors constructed a novel benchmark named REASONMAP featuring:

• Scope: 1,008 human-verified question-answer pairs derived from high-resolution transit maps of 30 cities across 13 countries.
• Data Pipeline:
  1. Collection & Preprocessing: 30 high-res maps were collected. MLLMs (e.g., GPT-4o) were used to extract line names and stops, followed by manual correction to handle transfer stations and branches. Data was unified into a JSON format ("Metro Data").
  2. Question Generation: Two question types were generated based on templates:
     • Short Questions: Ask for the route name, departure, and arrival stops.
     • Long Questions: Require identifying the number of via stops or listing every intermediate stop.
  3. Reference Routes: Valid routes were retrieved via Google Maps and Gaode Maps APIs, ensuring visual traceability on the provided maps.
  4. Difficulty Labeling: A two-level labeling system was applied:
     • Map Difficulty: Based on the number of lines and transfer stations (Easy, Medium, Hard).
     • Question Difficulty: Based on the number of transfers required in the route (0 transfers = Easy, 1 = Medium, >1 = Hard).
• Quality Control: Manual verification by domain experts ensured route correctness and visual consistency.

B. Evaluation Framework

A two-level evaluation framework was proposed to assess both correctness and quality:

1. Correctness Evaluation: Verifies if the departure/arrival stops match, route names exist in the Metro Data, and transfer points are consistent.
2. Quality Evaluation (Map Score): A unified scoring metric (0–20 for short questions, 0–40 for long questions) that rewards:
   • Correct endpoint matching.
   • Correct route name identification.
   • Accuracy in counting via stops (for long questions).
   • Intersection-over-Union (IoU) for listed intermediate stops.
   • Bonus points are awarded if the answer is fully correct.

C. Training Baseline

To address the non-differentiable nature of the rule-based evaluation metrics, the authors established a Reinforcement Fine-Tuning (RFT) baseline using Group Relative Policy Optimization (GRPO).

• Reward Design: Two components were used:
  1. Accuracy Reward: Binary signal based on correctness.
  2. Format Reward: Encourages parsable outputs adhering to the strict template.
• Goal: Optimize model behavior to align with the evaluation criteria, specifically tested in a cross-city setting (training on some cities, testing on disjoint cities) to measure generalization.

3. Key Contributions

1. REASONMAP Benchmark: A scalable, semi-automated pipeline for creating a diverse, high-resolution transit map dataset with fine-grained spatial reasoning tasks.
2. Two-Level Evaluation: A novel framework separating answer correctness from quality, providing a more nuanced assessment than simple accuracy.
3. Comprehensive Benchmarking: Extensive evaluation of 16 popular MLLMs (both open-source and closed-source, base and reasoning variants).
4. Training Baseline: A GRPO-based reinforcement learning setup that demonstrates performance improvements across model scales in cross-city scenarios.

4. Experimental Results & Key Findings

A. Performance Trends

• Open-Source vs. Closed-Source:
  • Open-Source: Counterintuitively, base models outperformed reasoning-tuned variants (e.g., Qwen2.5-VL-72B Base > QvQ-72B Reasoning). Reasoning models often engaged in excessive "trial-and-error" self-correction, leading to visual confusion and overriding correct initial findings.
  • Closed-Source: The opposite trend was observed; reasoning variants outperformed base models (e.g., OpenAI o3 > OpenAI 4o). These models demonstrated stronger visual grounding, allowing them to self-correct errors within the reasoning chain effectively.
• Scaling Law: Larger models generally achieved higher accuracy with fewer tokens, confirming that scaling laws hold for fine-grained visual reasoning.
• Visual Grounding Necessity: When visual inputs were masked (text-only input), performance dropped significantly for most models, especially closed-source ones. This confirms that successful performance relies on direct visual grounding rather than just leveraging internal knowledge priors.

B. Error Analysis

Common failure modes identified include:

• Visual Confusion: Misidentifying lines due to similar colors or adjacent layouts.
• Format Errors: Generating correct content but failing to adhere to the required output structure.
• Hallucination: Generating stops or lines not present in the image.
• Refusal: Models explicitly declining to answer complex spatial tasks.

C. Training Results

The GRPO-based reinforcement fine-tuning consistently improved both accuracy and map scores across different model scales (3B, 7B) in cross-city settings, proving that reward signals can effectively optimize spatial reasoning capabilities.

5. Significance

• Bridging the Gap: REASONMAP moves beyond coarse spatial tasks to evaluate the complex interplay between fine-grained OCR, visual grounding, and multi-step spatial planning.
• Insight into Reasoning Models: The findings challenge the assumption that "reasoning-tuned" models are universally superior, highlighting that for visual tasks, excessive internal deliberation can sometimes degrade performance if visual grounding is weak.
• Real-World Applicability: The benchmark simulates real-world navigation challenges, offering a pathway to develop MLLMs capable of assisting in autonomous driving, urban planning, and navigation for visually impaired individuals.
• Future Direction: The work establishes a foundation for agent-based planning and iterative reasoning in interactive environments, moving from static image analysis to dynamic spatial intelligence.

In conclusion, REASONMAP provides a rigorous standard for evaluating the next generation of multimodal models, emphasizing that true visual reasoning requires robust integration of visual perception and logical planning, rather than just textual inference.