🤖 AI

OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

OmniMapBench is a new benchmark comprising over 2,000 manually annotated question-answer pairs across diverse map documents, designed to evaluate and advance visual-centric reasoning in Large Vision-Language Models by introducing the Visual Dependency Index to measure irreducible visual grounding.

Original authors: Yang Chen, Yunwen Li, Yufan Shen, Minghao Liu, Tianyu Zheng, Bin Fu, Qunshu Lin, Zhi Yu, Botian Shi

Published 2026-07-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yang Chen, Yunwen Li, Yufan Shen, Minghao Liu, Tianyu Zheng, Bin Fu, Qunshu Lin, Zhi Yu, Botian Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to read a map. You might think, "No problem! Just have the robot read the text on the map, like a book." But here's the twist: maps are sneaky. They hide their secrets in lines, colors, and shapes that you can't just "read" with words.

This paper introduces a new challenge called OmniMapBench, which is basically a giant, tricky obstacle course made entirely of maps. The authors built this course to see if robots can actually see and think about what they see, or if they're just cheating by reading the text instead.

The Great "Text-Trap"

The paper argues that many previous tests for AI were like giving a student a math problem where the answer was hidden in the question's wording. If the robot could turn the picture into a list of words (like "70% of smokers feel..."), it could solve the puzzle without ever really looking at the image.

The authors say: "Stop cheating!" They argue that many existing tests are too easy because the pictures can be turned into text too easily. They explicitly rule out the idea that reading text descriptions is enough to solve map problems. In fact, they show that for some maps, if you try to describe the whole picture in words, you miss the crucial details needed to answer the question.

The New Challenge: A Map Jungle

To fix this, the team created OmniMapBench. Think of it as a massive library containing 1,603 different maps. These aren't just boring subway maps; they are a wild mix of:

Indoor navigation (finding a soda machine in a mall).
Topography (reading mountain heights and river paths).
Fantasy game maps (finding a building with an eye symbol).
Historical explorer routes (tracing a ship around Africa).

They hand-crafted 2,096 questions for these maps. Some are easy (just spotting a color), while others are super hard (requiring the robot to take three steps to figure out a route).

The "Visual Dependency" Test

How do you know if a robot is actually looking at the map or just guessing? The authors invented a clever test called the Visual Dependency Index (VDI).

Imagine you have a robot.

Test A: You show it the map and ask, "How many houses are on the right?" It answers.
Test B: You take the map away. Instead, you give the robot a long, boring paragraph describing the map in words (like "a green field with four clouds..."). Then you ask the same question.

If the robot gets Test B wrong, that's good! It means it needed the picture to solve the problem. The VDI measures exactly how much the robot's score drops when you take the picture away.

High VDI: The robot failed without the picture. It was actually looking!
Low VDI: The robot got it right even with just words. It was just reading the text.

The paper measured this and found that OmniMapBench has a much higher VDI than other tests. Even with a huge amount of text allowed to describe the image, the robots still struggled without the actual picture. This proves the test is fair and actually checks for visual skills.

The Results: Robots Are Still Learning

The authors put 25 of the smartest AI models (both free and paid ones) through this obstacle course.

The Best Score: The top robot, Gemini-3.1-Pro, got 75.03% correct.
The Gap: That might sound high, but in the world of AI, it means there's still a lot of room for improvement. The paper notes that even the best models struggle with the hardest, multi-step reasoning tasks.
The "Blind" Test: When they took away the pictures entirely and just gave the robots the questions and multiple-choice answers, the scores crashed from an average of 58.87% down to 23.35%. This proves the robots can't just guess based on the words; they really do need the map.

What's Next?

The paper doesn't claim this problem is solved. Instead, it suggests that current AI models are getting better at reading text but are still having a tough time with complex, visual reasoning. The authors hope that by using this new benchmark, researchers will build robots that can truly "see" the world, not just read the labels on it.

So, if you're a robot trying to find your way through a maze, don't just read the signs—look at the walls! The paper shows that right now, even the smartest robots are still learning how to do that.

Technical Summary: OmniMapBench

Problem Statement

Recent advancements in Large Vision-Language Models (LVLMs) have shifted the research focus from simple perception tasks (e.g., object recognition) to complex, visually grounded reasoning. However, the authors identify a critical limitation in existing document understanding benchmarks: much of the visual content in these datasets is reducible to text. Through Optical Character Recognition (OCR), layout analysis, or code generation, images of charts, tables, and structured documents can often be converted into textual representations (e.g., Markdown) that allow models to solve questions using linguistic inference alone, bypassing genuine visual grounding. This "textualization shortcut" obscures the true visual reasoning capabilities of LVLMs.

To address this, the paper argues that maps represent a document category that inherently resists textual reduction. Unlike structured documents, the semantic content of maps is embedded in spatial topology, symbolic conventions, and continuous graphical features (e.g., contour lines, color-coded regions) that cannot be exhaustively captured by linear text descriptions without prior knowledge of the specific query.

Methodology

1. OmniMapBench Dataset Construction

The authors introduce OmniMapBench, a benchmark specifically designed to foster visual-centric reasoning for map documents. The dataset construction follows a three-stage pipeline:

Data Acquisition: A corpus of over 5,000 map images was gathered via web scraping and manual collection from government portals and social media. Images were filtered for redundancy, resolution, and political sensitivity, resulting in 1,603 high-quality map images.
Annotation: Professional annotators created 2,096 manually verified question-answer (QA) pairs. Questions were formulated in objective formats (single-choice, multiple-choice, and sorting) to minimize ambiguity.
Cross-Validation: Two independent teams verified the QA pairs for clarity, correctness, and adherence to capability levels, discarding ambiguous or incorrect entries.

The dataset spans nine diverse categories: Indoor Navigation, Education & History, Engineering & Facilities, Transportation, Tourism, Topography & Remote Sensing, Environment & Nature, Socioeconomic, and Game maps. The QA pairs are categorized into three capability levels:

Level 1 (Perception): Basic information extraction.
Level 2 (Single-step Reasoning): Direct spatial reasoning.
Level 3 (Multi-step Reasoning): Complex, multi-step relational reasoning.

2. Visual Dependency Index (VDI)

To quantitatively measure a benchmark's reliance on visual information, the authors propose the Visual Dependency Index (VDI).

Definition: VDI measures the normalized accuracy drop when the full visual input is replaced by a text-only, question-agnostic description generated by a reference LVLM under a specific token budget ( $k_{max}$ ).
Formula: $VDI = \frac{Acc_{full} - Acc_{text}}{Acc_{full}}$
Interpretation: A high VDI indicates that the task is difficult to solve via textual shortcuts and requires genuine visual grounding. A low VDI suggests the task can be solved primarily through text.
Complementary Metric: The Descriptive Saturation Index (DSI) is introduced to measure the ratio of actual tokens used to the maximum budget, indicating the density of information required for description.

3. Evaluation Protocol

The benchmark was used to evaluate 25 leading LVLMs, including both proprietary (e.g., GPT-5, Gemini-3.1-Pro) and open-source models (e.g., Qwen3.5 series, InternVL). The evaluation employed a strict exact-match accuracy metric and utilized Chain-of-Thought (CoT) prompting to elicit maximum reasoning capabilities.

Key Results

1. Performance Gap

Comprehensive evaluations revealed a significant performance gap among current models.

The top-performing model, Gemini-3.1-Pro, achieved an accuracy of 75.03%.
The best open-source model, Qwen3.5-397B-A17B, achieved 72.22%, narrowing the gap with proprietary systems to less than 3 percentage points.
Performance generally declined as reasoning complexity increased, with Level 3 (Multi-step reasoning) tasks proving to be the primary bottleneck for all models.

2. Visual Dependency Analysis

The VDI analysis demonstrated that OmniMapBench possesses a significantly higher reliance on visual information compared to established benchmarks (e.g., DocVQA, ChartQA, AI2D).

At a token budget of $k_{max} = 1024$ , OmniMapBench maintained a VDI of 0.338, the highest among all evaluated benchmarks.
In contrast, benchmarks like AI2D and MMBench dropped to near-zero VDI (0.019 and 0.028, respectively) at the same budget, indicating their tasks are largely solvable via text.
No-Image Baseline: When models were tested without any visual input (text-only), average accuracy dropped from 58.87% to 23.35%, confirming that the benchmark cannot be solved through language priors or answer-choice bias alone.

3. Category Variance

Performance varied significantly across map categories. Models performed better on categories with structured textual information (e.g., Tourism) and struggled with categories requiring the interpretation of dense, non-standard symbols and complex topological relationships (e.g., Topography & Remote Sensing).

Significance and Contributions

The paper claims three primary contributions:

OmniMapBench: The introduction of a diverse, visual-centric benchmark featuring 2,096 manually annotated QA pairs across 9 map categories, designed to probe a hierarchy of skills from perception to multi-step reasoning.
Visual Dependency Index (VDI): The proposal of a simple yet effective metric to quantify the extent to which a benchmark relies on visual information versus textual shortcuts.
Comprehensive Evaluation: A systematic evaluation of 25 LVLMs that identifies performance bottlenecks and highlights the challenges current models face in visual-centric reasoning.

The authors state that the work aims to catalyze progress in visual-centric reasoning for document understanding. By providing a benchmark that resists textual reduction and a metric to quantify this resistance, OmniMapBench offers a principled framework for assessing and improving the integration of perception and reasoning in multimodal models. The dataset and code are made publicly available to support this goal.