Technical Summary: PlanBench-V

Problem Statement

Spatial planning maps are critical instruments in territorial governance, translating regulatory frameworks and strategic objectives into visual representations. However, interpreting these maps requires a complex synthesis of fine-grained visual perception, spatial reasoning, and policy-informed professional judgment. While Vision-Language Models (VLMs) have advanced in general multimodal tasks, they face significant limitations in specialized domains like urban planning. Existing benchmarks primarily focus on general visual understanding or basic geographic interpretation, failing to capture the domain-specific cognitive processes required for statutory planning map analysis. Current VLMs struggle with recognizing specialized cartographic elements (e.g., legends, zoning boundaries), parsing complex spatial relations, and, crucially, applying normative planning knowledge to evaluate and optimize proposals. There is a lack of systematic evaluation frameworks to assess VLMs in these professional contexts, hindering the development of intelligent planning support systems.

Methodology

1. Dataset Construction: Spatial Planning Map Database (SPMD)

The authors constructed a high-quality, expert-annotated dataset comprising 223 planning maps and 1,629 question-answer (QA) pairs.

• Sources: The maps were curated from three sources to ensure diversity and accuracy: (1) Official Master Plans from China; (2) The Chinese Certified Urban-Rural Planner Qualification Examination (CURPQE); and (3) International planning maps from Japan, the UK, the US, the Netherlands, Canada, Germany, Singapore, and Mexico.
• Annotation: Professional urban planners manually formulated QA pairs to target four progressive cognitive dimensions. Reference answers were generated via a hybrid process: initial generation by Qwen2.5-VL-72B-Instruct, followed by rigorous review, correction, and rewriting by human experts to ensure factual correctness and minimize hallucination.
• Preprocessing: Manual screening was conducted to remove extraneous visual elements (icons, photos), and low-resolution maps were enhanced to ensure text legibility.

2. Evaluation Framework

The paper proposes a four-dimensional evaluation framework reflecting the cognitive pipeline of planning practice:

• Perception: Assesses the identification of visual components, including layout configurations, textual annotations, geographic features, and drawing elements. Subtasks include Element Recognition and Captioning.
• Reasoning: Evaluates the derivation of structured insights from visual and semantic information. Subtasks include Classification (identifying map types like master plans vs. detailed plans), Spatial Relationship Reasoning (topological, ordinal, and metric relations), and Domain-Specific Reasoning (analyzing spatial layout, functional organization, transportation, and ecology).
• Association: Tests the ability to link visual elements with relevant background policies, regulations, and planning indicators (e.g., zoning laws, ecological redlines).
• Implementation: Addresses the capacity for comparative critique and optimization of planning proposals. Subtasks include Scheme Evaluation (assessing strengths/weaknesses) and Decision-making (selecting optimal directions under constraints).

3. Experimental Setup

• Models Evaluated: Two rounds of evaluation were conducted across 17 VLMs.
  • Round 1 (2025): 11 models including GPT-4o, GPT-4o-mini, and various open-source models (Qwen-VL, InternVL) ranging from 2B to 78B parameters.
  • Round 2 (2026): 6 models incorporating enhanced agentic reasoning capabilities (e.g., Qwen3.6-Plus, Gemini-2.5-Pro, GPT-5.4).
• Evaluation Protocol: All models were tested using their original weights without fine-tuning. Evaluation utilized an LLM-as-a-judge framework with structured scoring rubrics and automated prompt optimization to handle subjective planning tasks.
• Validation: Human performance baselines were established using graduate students, and judge calibration confirmed that structured rubrics allow smaller models (e.g., GPT-4o-mini) to achieve evaluation fidelity comparable to larger judges.
• Sensitivity Analysis: A resolution sensitivity analysis confirmed that image resolution variations (up to 46-fold) did not introduce systematic bias in model rankings.

Key Results

Performance Trajectory

• Generational Leap: The best 2026 model, Qwen3.6-Plus, achieved an overall score of 1.701, outperforming the best 2025 model, GPT-4o (1.342), by a margin of 27%. Notably, even the lightest 2026 model (Qwen3.6-Flash) surpassed the top 2025 score, indicating a substantive generation-level uplift driven by agentic reasoning.
• Dimensional Gains: Improvements were uneven across dimensions. The most significant gains were observed in Scheme Evaluation (+77.7%), Spatial Reasoning, and Decision-making. Smaller gains were seen in Captioning and Policy Association.

Specific Findings

• Perception: Large models generally performed well in element recognition, but persistent issues included color misclassification and hallucination of objects in crowded scenes.
• Reasoning: Models struggled with multi-step logic and condition-based classification. While GPT-4o led in the first round, 2026 models showed improved capacity for structured inference.
• Association: Even top models exhibited limited policy awareness, often failing to distinguish between administrative levels or functional classifications of regulations, leading to conceptual overlaps.
• Implementation: This remained the most challenging dimension. Models frequently produced verbose, unfocused outputs lacking prioritization or value judgments. They often failed to identify critical constraints (e.g., zoning incompatibilities) and struggled to generate actionable, terminology-grounded advice.

Significance and Claims

The paper claims to introduce PlanBench-V, the first comprehensive benchmark designed specifically to evaluate VLM performance in spatial planning map interpretation. Its significance lies in:

1. Bridging the Gap: Providing a rigorous, expert-annotated dataset and evaluation framework that moves beyond general visual understanding to domain-specific professional judgment.
2. Revealing Limitations: Highlighting that while agentic reasoning drives substantial progress, current VLMs still lack the deep policy sensitivity and constraint-aware decision-making required for professional planning contexts.
3. Guiding Future Development: The findings suggest that general-purpose reasoning enhancements are insufficient for planning tasks. The authors argue for the necessity of domain-adaptive multimodal learning strategies that explicitly encode planning logic, regulatory constraints, and spatial decision-making processes to bridge the gap between general intelligence and expert-level urban planning cognition.

The paper concludes that PlanBench-V serves not only as an evaluation tool but also as a conceptual foundation for advancing VLMs toward reliable applications in spatial governance.